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SOME ISSUES CONCERNING THE USE OF DIDACTICS OF BIOLOGY 


Rita Birzina 
University of Latvia, Latvia 


Teaching biology is interesting and challenging, at the same time, it is also a dilemma because new and new 
advances are being made in biological science: biodiversity is changing - new species are being discovered while 
others are threatened with extinction, and unprecedented genetically modified organisms are being created. 
Teachers find it difficult to keep up with all the latest developments, especially in cell and molecular biology (Tun- 
nicliffe & Ueckert, 2007). Biology is a unique science that studies living things. The discipline of biology differs from 
other fields of science (e.g. physics and chemistry) in the breadth and complexity of its knowledge content and the 
interconnectedness of knowledge at many different levels (Wandersee et al., 2000). Regarding the use of different 
research methods, biology is also involved in exploring both the micro-world through microscopy, experimental 
work in the laboratory or in field studies of the macro-world. 

Biological knowledge encompasses a range of concepts with varying degrees of complexity, abstraction and 
meaning (Zogza, 2016), so the level of background knowledge of learners is important (Birzina et al., 2019; Cedere 
et al., 2022; Sorgo & Siling, 2017; Machar et al., 2014). Successful learning in biology requires the use of diverse 
teaching/learning methods not only for understanding the natural world, but also for learners’ successful engage- 
ment in today’s labour market through project-based learning, critical thinking, problem solving, civic engage- 
ment, research and digital skills (Birzina et al., 2021; Cedere et al., 2020; Mard & Hilli, 2022). There are so many new 
developments in life sciences that nowadays previous theoretical constructs - theories, laws and models - need to 
be revisited and mastered at the school level. In these new circumstances, the highest priority should be given to 
improving the didactics of biology so that students can acquire biological knowledge and understand biology as 
a science, avoid misconceptions, learn according to the digital age and use an interdisciplinary approach in their 
learning (Tsybulsky, 2020). 

While biology and natural sciences share many aspects, the focus of biology on life creates unique philo- 
sophical, methodological and ethical preconditions for understanding biology (Kloser, 2012). This means that the 
study of the teaching and learning process in the biological sciences is a pressing issue and this area of research is 
called ‘biology didactics’ and aims to highlight and facilitate the process of teaching and learning about the bio- 
logical world. Since the biological world is part of the natural world and biological sciences are part of the natural 
sciences, then ‘biology didactics’ can only be considered as part of the research area of ‘science didactics’ (Zogza, 
2016, cited in Lewis, 2008). 

To understand the nature of the didactics of biology, it is necessary to clarify its place in education. There are 
two didactic approaches known: the Anglo-American and the European (Meyer, 2012; Ligozat & Almqvist, 2018). 
In Anglo-American education, the view of pedagogy as a science, often borrowed from Herbart, spread in the late 
19th century (Hamilton, 1999). In the Anglo-American approach, the laws of psychology ground the teaching, 
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while teaching methods are primarily a matter of subject-teaching strategy. In the European approach, didactics 
develops as a theory and methodology of education and learning and is part of pedagogy (Zogla, 2001), which 
focuses more on strategies, methods and different approaches to teaching and learning. 

Two areas of didactics must be distinguished: general didactics and subject didactics. In European countries, 
there is a heterogeneous approach to understanding didactics and subject didactics: some subject didactics are 
considered subfields of general didactics (Krogh & Qvortrup, 2021) because of the relationship between general 
didactics and subject didactics. In Sweden, for example, the term general didactics was introduced in the late 1980s 
to make connections between general didactics and subject didactics (Ligozat et al., 2015) and pedagogy as an 
academic discipline has a strong influence on educational policy, while subject didactics is a relatively independent 
academic discipline. In Denmark, subject didactics is less developed than in Sweden or Norway (Zogla, 2001). In 
Spain, didactics is a scientific-pedagogical discipline that addresses each stage of the teaching and learning process 
(Vergara, 2015). In general, the term “subject didactics” is used in Scandinavian, German, Swiss, French, Spanish 
and other European countries. The term “general didactics” is used less frequently (in Finland, Norway, Denmark, 
and other countries). The use of didactics does not exist in the UK, although some aspects of didactics do appear, 
such as curriculum theory and research on the teaching/learning process (Meyer, 2012). 

Since the 1980s, under the influence of globalisation and the importation of Anglo-American educational 
paradigms, the state of didactics and the relationship between general and disciplinary didactics has been an 
increasingly contentious issue in both the academic world and the field of teacher education (Ligozat & Almaqvist, 
2018). On the one hand, subject didactics claims the status of an independent science, while at the same time, it 
is part of general didactics and general didactics, in turn, is part of pedagogy. Thus, subject didactics integrates 
parts of the respective subject, curriculum, pedagogy, didactics, psychology, communication theory and practice, 
becoming an independent science within the system of pedagogical sciences (Zogla, 2001) including a highly 
structured and grounded approach to teaching and learning. The analysis of European understandings of didactics 
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also shows that the understanding of didactics as a science is multi-coloured, multi-faceted and diffuse (Meyer, 
2012). The relationship between teaching and research is often accepted and just as often ignored. In fact, research 
should and does influence teaching (and vice versa), but the gap between them sometimes is wide (Kubiatko, 2012). 
Ideally, didactics should simultaneously cover three levels: (1) the theoretical or research level, where didactics 
denotes the field of research; (2) the practical or implementation level, where didactics mainly involves teaching 
and learning, curriculum design and the school learning process; and (3) the discursive level, where didactics shapes 
the professional dialogue by discussing teaching and learning issues (Hopmann & Gundem, 1998). 

In order to identify trends in the understanding of biology didactics in the scientific literature, a systematic 
review was performed by conducting quantitative and qualitative research on the use of biology didactics in the 
Web of Science and SCOPUS databases. The keywords “didactics’, “biology” and “biology didactics” or “didactics of 
biology” were selected for the information search. 

The limiting criteria were the publication period of the articles used in the systematic review (1990-2023), lan- 
guage (English), publication only in scientific journals, selected databases (Web of Science and SCOPUS), excluding 
articles that are in both databases (n = 5) and full-text availability (n = 1). Figure 1 illustrates the selection of articles. 

As can be seen in Table 1, all the articles analysed have reached the research, practical and discursive levels of 
biology didactics. At the research level, a study is usually carried out and/or the scientific literature on the research 
question is systematised. At the practical level, research is carried out on the pupil's, student’s or teacher's activ- 
ity related to the use of a particular teaching method or technique. For example, the project method (Kubiatko 
& Vaculova, 2011), making vignettes (Sanders, 2022), identification of misconceptions and analysis of teaching/ 
learning materials (Angosto Sanchez & Morcillo Ortega, 2022), developing word associations and exploring out- 
door experiences (Schmding & Grotjohann, 2021). For feedback and professional dialogue, there are workshops 
organised (Kubiatko & Vaculova, 2011), project work (Machar et al., 2014; Schmaing & Grotjohann, 2021), learning 
and sharing takes place in the form of different courses (Machar et al., 2014; Sanders, 2022), new methodological 
materials and learning models (Angosto Sanchez & Morcillo Ortega, 2022), and courses are developed (Machar et 
al., 2014). It should be noted that the didactic approach to the acquisition of biology is mostly seen in the context 
of cross-subject links and projects are usually more interdisciplinary, making links between biology and other 
science subjects of chemistry, physics and geography, at the same time also including social sciences, as well as 
environmental and health education. 


Table 1 
Research, Practical and Discursive Level in Understanding the Didactics of Biology 


Authors 


Angosto Sanchez & Morcillo 


Ortega, 2022 
(Spain) 


Kubiatko & Vaculova, 2011 


(Czech Republic) 


Machar et al., 2014 (Czech 


Republic) 


Sanders, 2022 
(Sweden) 
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AS 


Research level 


Questionnaires of students from four 
different academic levels gathering 
the ideas about plant nutrition and 
photosynthesis 


Pupils’ practical activities in the project 
are explored 


The survey of 119 graduates of Mas- 
ter’s courses in Teacher Education in 
Biology to determine their opinions on 
the importance of biology and environ- 
mental education didactics during their 
pre-graduate studies 


Multimodal objects are considered as 
boundary objects that can facilitate 
learning conversations, both cognitive 
and affective 


Practical level 


Analysis of learning misconceptions 
and analysis of textbooks and university 
manuals 


Project-based learning, implementing 
the interdisciplinary approach takes 
place 


Evaluation of the content and benefit of 
the completed courses of biology and 
environmental education didactics in 
biology studies with respect to its signifi- 
cance for the current teaching practice 


Three vignettes are used to make public 
an individual teacher's reflections on 
teaching situations in three courses 
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Discursive level 


Suggestions on how to improve the 
teaching/ learning process are offered 
-a teaching model of plant nutrition 
and photosynthesis 


Discussion between the teacher 
and pupils 


Discussion of the feedback received 
from participants in the context of cur- 
rent changes in the education system 
and discussion about subject-matter 
didactics of biological disciplines 


Practice-based reflective narratives 
on specific teaching moments with 
multimodal objects, which can sup- 
port pupils’ engagement in biology 
didactics and related classes 
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Authors Research level Practical level Discursive level 


Schmaing & Grotjohann, Guided interviews, documents such Pupils develop word associations using _It is implemented as a project of the 


2021 (Germany) as concept maps can be analysed, _ stimulus words “Wadden Sea’, “mudflat biology didactics with the aim of im- 
questionnaires can be used, or field hiking tour’, and “tides”. plementing the studies of the biology 
observations can be made Collection and analysis of experiences _ didactics that have a concrete focus on 


(school, out-of-school with the class, the Wadden Sea as a place of learning 
and private) 


The selection of articles shows that the relationship between research and the teaching/learning process 
is interconnected: both research is linked to the teaching/learning process and the teaching/learning process is 
linked to research. Biology didactics provides a holistic view of the biology learning process, taking into account the 
key actors (teachers, pupils, students, teaching/learning objectives, the subject content and structure of biology) 
and their different relationships in the didactic triangle, thus incorporating the core principles of general didactics 
(Kubiatko, 2021; Lampiselka et al., 2019). Overall, it can be concluded that in the context of the contemporary 
educational paradigm shift (Krogh & Qvortrup, 2021) and because of synergies with general didactics (Ligozat & 
Almavist, 2018), the didactics of biology is becoming stronger and more visible, and it shows itself as “a dynamic 
discipline” (Kubiatko, 2021) and continues to develop as an independent science. 
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ANXIETY AND ATTITUDES OF 
MIDWIFERY STUDENTS TOWARD 
SCIENTIFIC RESEARCH 


Seyda Ferah Arslan 


Introduction 


For a profession to gain a scientific quality, its members should detect 
the challenges they confront, identify the problem and the variables associ- 
ated with it, and solve these problems utilizing scientific research techniques 
(Munir & Bolderston, 2009). Midwifery research improves the practice of mid- 
wifery and its evidence-based practices (Luyben et al., 2013). Midwives should 
have a positive attitude toward scientific research to implement evidence- 
based practices. Unfortunately, clinical midwives do not engage enough in 
research and cannot reflect the outcomes of treatment procedures (GUner 
et al., 2015). Undergraduate education is an effective process for gaining a 
researcher's identity (Clark et al., 2009). The attitudes of midwifery students 
toward doing scientific research are determining factors in increasing the 
quality of midwifery and strengthening evidence-based practices. 


Background 


Midwifery training is given in Turkiye at bachelor, master, and doctoral 
levels with international standards. Within the scope of the Bologna process, 
standardization has been made in midwifery education programs in paral- 
lel with other higher education programs. It is expected that the midwife 
candidates who complete the midwifery program have sufficient knowledge 
and skills specific to the field, especially in professional ethics, occupational 
legislation, and conducting scientific research (Council of Higher Education, 
2016). Midwifery education was designed to support reading, critique, un- 
derstanding and conducting research skills for midwifery students. 

For a profession to gain scientific quality, it is necessary to find scientific 
information content produced by scientific research and to transfer this infor- 
mation to practice (Campasi & Finn, 2011; Keib et al., 2017; Kes & Sahin, 2019). 
Midwifery, which is a professional discipline, has knowledge accumulated by 
all its members. Conducting research to increase professional knowledge 
is one of the essential elements of a profession (Munir & Bolderston, 2009). 
Basing midwifery practices on scientific knowledge strengthens the profes- 
sional identity of the profession. Midwifery research is a way to improve the 
practice of midwifery and its evidence-based practices (Halabi, 2016; Luyben 
et al., 2013; Unver et al., 2018). 
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Abstract. The attitudes of midwifery 
students, who are being trained to perform 
the profession of midwifery in the future, 
toward doing scientific research are 
determining factors in increasing the 
quality of midwifery and strengthening 
evidence-based practices. Undergraduate 
education is an important stage in which 
basic research-related perspectives and 
skills are gained for health professions. 

This descriptive study aims to determine 
the anxiety and attitudes of midwifery 
students toward scientific research and to 
examine the variables that affect them. 

It was conducted at a public university 

in western Turkiye and included 246 
undergraduate midwifery students. 
Midwifery students’ anxiety toward 
scientific research was found to be low 
and their attitudes toward scientific 
research were positive. The attitudes of 
students who have experience in doing 
scientific research, participated in scientific 
congresses/symposia, have the desire to do 
scientific research under the guidance of 
aconsultant and follow periodicals, were 
found to be more positive. Anxiety and 
positive attitude toward scientific research 
were found to be negatively related. The 
results of the regression analysis showed 
that the students’ Research Anxiety affected 
their Attitude Toward Scientific Research. 
Students’ experiences of conducting 
scientific research reduce anxiety and 
affect attitudes positively. Therefore, 
undergraduate students should be actively 
involved in scientific research and activities. 
Keywords: Health professions education; 
midwifery student; research anxiety; 
research attitude 
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Itis important for midwives to conduct research and use the information obtained as a result of their research 
to develop knowledge and applications specific to midwifery. However, studies are mostly carried out by academic 
midwives. Conversely, clinical midwives do not engage enough in research and cannot reflect the outcomes of 
treatment procedures (Gunner et al., 2015). All over the world, studies in the field of midwifery are increasing in 
number and quality. Universities are expected to educate students about scientific attitudes and behaviours as 
well as to lead scientific research (Keib et al., 2017; Unver et al., 2018). Midwifery students must meet the research 
culture during their undergraduate education and adopt positive attitudes toward scientific research in order to 
develop the midwifery profession, contribute to its professionalization, and increase the quality of care, autonomy, 
and power of midwives. 

Undergraduate education is an important stage in which basic research-related perspectives and skills are 
gained. It is an effective process in gaining researcher identity (Clark et al., 2009). Scientific research education 
develops not only knowledge and skills but also an attitude toward research. The first step in ensuring the ef- 
fectiveness of research begins with the research training that its members receive as part of their undergraduate 
education (Finkelman & Kenner, 2013; Tsai et al., 2014). It is expected that research education provided at the 
undergraduate level positively affects students’ attitudes and anxieties toward conducting research. A high level 
of research anxiety reduces the sense of self-efficacy and causes a negative attitude toward research (Lei, 2008). 
Undergraduate education is especially important for midwifery students because it allows them to become ac- 
quainted with areas of research in midwifery and to develop research-specific values and skills. There are scientific 
research methods and statistics courses in the midwifery undergraduate curriculum. These courses are designed to 
motivate students to conduct midwifery research, teach research stages, and increase their willingness to take part 
in research projects. However, in Turkiye, scientific research education mostly seems to be necessary for graduate 
levels rather than undergraduate level education. To ensure that competent graduates are ready to safely care 
for patients, evidence-based education, which incorporates research to drive instructional methods, is essential 
(Oermann & Kardong-Edgren, 2018; Ross & Burrell, 2019). 


Aim and Research Questions 


Overall, few published studies have examined the attitudes of undergraduate midwifery students regarding 
research. No study has been found in Turkiye that examines midwifery students’ anxiety and attitudes toward sci- 
entific research. The attitudes of midwifery students, who are being trained to perform the profession of midwifery 
in the future, toward doing scientific research are determining factors in increasing the quality of midwifery and 
strengthening evidence-based practices. The purpose of this study was to ascertain the attitudes and anxiety levels 
of undergraduate midwifery students regarding scientific research and to evaluate those attitudes and levels in 
relation to various variables. 

For this purpose, answers to the following questions were sought: 

1. What are the anxiety and attitudes levels of midwifery students toward scientific research? 

2. Domidwifery students’anxiety and attitudes levels toward scientific research show a statistically signifi- 
cant difference based on their study year, take a statistics or research methods course, scientific research 
experience, follow periodicals in the field of health sciences, read and examine articles, participate in 
scientific congress/symposium and want to do scientific research variables? 

3. Is there a significant relationship between midwifery students’ anxiety and attitudes levels toward 
scientific research? 

4. |s midwifery students’ anxiety a significant predictor of attitudes toward scientific research? 


Research Methodology 
General Background 
The current study was planned as a descriptive and cross-sectional investigation. This study was conducted 
at a university in western Anatolia, Turkiye, in the midwifery department. The data were obtained from the survey 


process in May 2021. The questionnaire was sent to the students online, and volunteer students were permitted 
to participate. 
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Participants and Procedure 


The research group included 319 students enrolled in the Midwifery Department during the academic year 
2020-2021. According to the sample calculation with the known population, it was decided that at least 174 students 
should be included in the sample (95 % confidence interval, 5 % margin of error) (Cochran, 1977). A convenience 
sampling method was used. Accordingly, 246 students accepted to participate in the study and completed the 
sample of the study (nearly 77% of the 319 students). 

Table 1 lists the general characteristics of the study participants. Students who studied in the third grade 
were 29.3%. Of the students who took statistics class were 20.3%, whereas 15.9% of them took research methods. 
Students who do not have any scientific research experience were 74.4%. Those who follow periodicals in the field 
of health sciences were 34.6%. Students who read and examined articles were 88.6%. Those who attended scientific 
congresses/symposia amounted to 33.7%. Students who wanted to do scientific research under the guidance of 
a consultant were 67.9%. The average age was 20.58 + 2.67. 


Table 1 
Descriptive Features of Students 


Range M SD 
Age 18-40 20.58 2.67 
n % 
1 58 23.6 
2 70 28.5 
Study year 
3 72 29.3 
4 46 18.7 
Yes 50 20.3 
Took a statistics course 
No 196 79.7 
Yes 39 15.9 
Took a research methods course 
No 207 84.1 
Yes 63 25.6 
Scientific research experience 
No 183 74.4 
Yes 85 34.6 
Follow periodicals in the field of health sciences 
No 161 65.4 
Yes 218 88.6 
Read and examine articles 
No 28 11.4 
Yes 83 33.7 
Participation in scientific congress/symposium 
No 163 66.3 
Yes 167 67.9 
Want to do scientific research under the guidance No 25 10.2 
of a consultant 
Hesitant 54 22.0 
Note. N= 246. 
Study Instrument 


The Research Anxiety Scale, the Attitude Toward Scientific Research Scale, and a 9-item questionnaire created 
by the researcher in response to the literature to gather data on the sociodemographic traits of the participants 
were used to gather data. 
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Research Anxiety Scale (RAS): it was created by BuyUk6zttirk to determine the students’ research concerns. The 
5-point Likert-type scale includes 12 items and a single factor. Expressions that directly reflect the state of anxiety 
(items 1,5, 6, 7,9, 10, and 12) were scored by giving numerical values from 5 to 1 from “completely agree” to “totally 
disagree.’ Items that do not directly reflect the state of anxiety (items 2, 3, 4, 8, and 11) were scored inversely. The 
scale yields a score that ranges between 12 and 60. Higher scores imply a high level of anxiety. The scale’s internal 
consistency coefficient is .870 (BuyUk6zturk, 1997). It was found to be .920 in this study. 

Attitude Toward Scientific Research Scale (ATSRS): Korkmaz et al. created it to assess students’ attitudes toward 
scientific research. It consists of 30 items on a 5-point Likert scale with four subscales. The increase in scores indi- 
cates a negative attitude for the first sub-dimension (Reluctance to Help Researchers) and second sub-dimension 
(Negative Attitude Toward Research), and a positive attitude for the third sub-dimension (Positive Attitude Toward 
Research) and the fourth sub-dimension (Positive Attitude Toward Researchers). A negative attitude is expressed 
by the first two subscales, while a positive attitude is expressed by the third and fourth subscales. For this reason, 
it is not meaningful to calculate a total score for the whole scale, and separate processing is required on sub- 
dimensions. Internal consistency coefficients were .85, .81, .80, and .76, respectively (Korkmaz et al., 2011). In this 
study, they are .871, .884, .924, and .947, respectively. 

Student Information Form: The researcher designed it in accordance with the literature (Kes & Sahin, 2019; 
Unver et al., 2018; Yilmaz et al., 2020). The characteristics of the students such as their age, gender, study year, 
whether they took statistics and research courses, their research experience, their experience of attending scientific 
meetings, and their status of following periodicals were asked. 


Data Analysis 


Categorical data were expressed with frequency and percentage distribution, whereas continuous data were 
expressed with arithmetic mean and standard deviation. The compliance of the data to normal distribution was 
examined using the Kolmogorov-Smirnov test, histogram, and probability plots. T-test and one-way ANOVA test were 
used in the comparison of the groups. Post Hoc analysis was performed according to the distribution of variances 
using Tukey or Tamhane tests. Correlation coefficients and statistical significance were calculated using the Pearson 
Correlation test. The significance level was accepted as p < .05. SPSS 21.0 statistics program was used for analysis. 


Ethical Considerations 


The organization where the study was conducted granted permission and ethics committee approval (protocol: 
02/28). Students who agreed to participate in the study provided written consent, which was obtained after they 
were informed. From the creators of the scales used in the study, necessary permissions were obtained. 


Research Results 


Students’ Research Anxiety: In Table 2, it is shown whether the “Attitude Toward Scientific Research Scale” scores 
of the students differ according to various variables. The anxiety level of students who took the research methods 
course (p = .033) had the experience of doing scientific research (p = .001), followed the periodicals in the field of 
health sciences (p = .020), and had the experience of participating in a scientific congress/symposium (p < .001) 
was significantly lower. The anxiety level of students who wanted to do scientific research under the guidance of 
a counsellor was found to be significantly lower than those who did not want to and who were hesitant about 
this (p = .002). Students’ level of anxiety did not change according to their grade level, taking statistics courses 
and reading/reviewing articles (p > .05). There was no relationship between the age of students and their anxiety 
toward scientific research (r= —0.119, p > .288). Students got an average score of 23.92 + 9.29 on the anxiety scale. 
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Table 2 
Students’ Scores of the Research Anxiety Scale 


M SD Test value p 
1 24.55 9.83 
2 24.42 10.48 
Study year F=.104 957 
3 23.67 7.15 
4 23.06 10.84 
Yes 23.19 9.83 
Took a statistics course t= .411 682 
No 24.16 9.18 
Yes 20.60 5.38 
Took a research methods course t=.213 .033* 
No 24.64 9.82 
Yes 19.45 5.59 
Scientific research experience t= .499 .001* 
No 25.50 9.85 
Yes 21.26 7.01 
Follow periodicals in the field of health sciences t= 366 .020* 
No 25.72 10.25 
Yes 23.60 9.20 
Read and examine articles t= .045 299 
No 27.43 10.34 
Yes 20.09 5.88 
Participation in scientific congress/symposium t= .632 < .001* 
No 26.52 10.29 
Yes 21.94b 8.27 
Want to do scientific research under the guidance No 32 Ada 11.96 F= 006 002" 
of a consultant ———— 
Hesitant 28.57a 8.86 


Note. N = 246. Means with different subscripts differ at the p = .05 level by Tukey’s HSD. 
*p <.05.**p < .001. 


Students’ Attitude Toward Scientific Research: Table 3 shows whether the scores of the students’ attitudes 
toward the scientific research scale differ according to various variables. The “Positive Attitude Toward Research” 
(p = .002) and“Positive Attitude Toward Researchers” (p = .019) sub-dimension scores of the students with scientific 
research experience were significantly higher than those who did not have that experience. Students who followed 
periodicals had a more positive attitude in all sub-dimensions than those who did not. Significantly lower scores 
were obtained in the “Reluctance to Help Researchers” (p = .010) and“Negative Attitude Toward Research’ (p = .006) 
sub-dimensions, whereas they got significantly higher scores in the “Positive Attitude Toward Research” (p < .001) 
and “Positive Attitude Toward Researchers” (p = .005) sub-dimensions. Students who read/examined articles got 
significantly higher scores from the “Positive Attitude Toward Researchers” sub-dimension (p = .024). Students 
participating in a scientific congress/symposium received significantly lower (p = .001) scores from the “Negative At- 
titude Toward Research” sub-dimension, whereas they got significantly higher scores in the “Positive Attitude Toward 
Research” (p < .001) and “Positive Attitude Toward Researchers” sub-dimensions (p = .006). Students’ attitudes did 
not change according to their grade level, or their status of taking statistics, or research methods courses (p > .05). 
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Table 3 
Students’ Scores of the Attitude Toward Scientific Research Scale 


Reluctance to be helpful Negative attitude Positive attitudes toward Positive attitude toward 
to researchers toward research research researchers 


M SD Test M SD Test M SD Test M SD Test 


1 7.21 571 16.53 5.28 26.16 5.37 25.78 5.08 


2 1854 558 F=1127 1681 534 F=2.025 2450 549 F=1257 2490 550 F=3.059 


Study year 
3 1849 588 p=.339 1869 560 p=.111 2504 605  p=.200 23.14 556 p=.059 


4 19.24 6.40 17.22 6.26 26.04 5.17 25.17 4.65 


Yes 18.76 6.09 t.565 1688 622 t=-0.696 2668 543 t=1916 2514 4.59 t= .738 
Took a statistics course 


No 1823 582 p=5/73 1750 546 p=487 2499 558 p=.057 2452 551 p=.461 


Yes 1849 6.09 t=.169 1603 606 t.634 2664 558 t=.975 25.03 5.26 t= .488 
Took a research methods 


course 
No 18.31 584 p=866 1763 551 p=.102 2509 556 p=.112 2457 536 p=.626 
Yes 1833 676 t=-0.013 16.57 546 t=-1.316 27.24 485 3192 2600 414 12.363 
Scientific research 
experience 


No 1834 554 p=.990 1765 566 p=.189 2468 568  p=.002* 2417 562 p=.019* 


Yes 17.02 629 t=-2591 16.04 512 t=-2751 2727 482 4287 2584 407 t=2866 
Follow periodicals in the 


field of health sciences 
No 19.04 552 p=.010* 1808 5.76 p=.006* 2432 570 p<.001* 2401 581 p=.005* 


Yes 1846 596 t=.909 1734 5.61 


- 
I 
oO 
DO 
Ww 
Oo 


25.50 545 171.240 2492 508 t=2276 
Read and examine 


articles 


No 1739 5.09 p=.364 1761 579 8 p=.816 2411 649 p=.216 2250 674 p=.024* 


Yes 1743 603 t-1.740 15.72 528 t=-3357 27.11 443 13.987 2595 408  t=2784 
Participation in scientific 


congress/symposium 


No 1880 574 p=.083 1821 561 p=.001* 2444 590 p<.001* 23.98 5.77  p=.006* 


Yes 5.96 F=5.450 1683b 562 F=3.506 2634a 5.56 F=10.556 25.08a 534 F=3.737 


Want to do scientific 
research under the guid- | No 19.28 588 p=.005* 17.24 621 p=.032* 21.68 7.00 p<.01* 22.00b 751  p=.025* 
ance of a consultant 


Hesitant 5.01 19.13a 5.06 23.93 3.67 24.50 3.60 


Note. N = 246. Means with different subscripts differ at the p = .05 level by Tukey’s HSD or TamhaneT2 test. *p < .05. **p < .001. 


Participants’ age and the sub-dimension “Reluctance to Help Researchers” showed a low level of significant 
negative correlation (r=.161, p = .012). As age increased, the score for reluctance to help researchers decreased. A 
low level of positive correlation was found between the age of the participants and the “Positive Attitude Toward 
Research” sub-dimension (r= .191, p = .003). As age increased, the positive attitude toward research increased. 
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The Relationship between the Research Anxiety and the Attitude Toward Scientific Research: The correlation 
findings between the two scales are given in Table 4. Significant relationships were found between the “Anxiety 
Toward Scientific Research Scale” and all sub-dimensions of the “Attitude Toward Scientific Research Scale.’ At 
moderate and high levels significant relationships were found between the sub-dimensions of “Reluctance to 
Help Researchers” (r = .517, p < .001) and “Negative attitude toward research” (r = .606, p < .001) of the attitude 
scale and anxiety scale. Increasing anxiety and negative attitude toward research increased the reluctance to 
help researchers. There was a negative, moderate to high significant relationship between the sub-dimensions of 
“Positive Attitude Toward Research” (r = —0.721, p < .001) and “Positive Attitude Toward Researchers” (r = —0.431, 
p < .001) of the attitude scale and anxiety scale. Increasing anxiety toward research decreased positive attitudes 
toward research and researchers. 


Table 4 
The Relationship between the Research Anxiety Scale and the Attitude Toward Scientific Research Scale 


Attitude Toward Scientific Research Scale 


Reluctance to be Negative attitude Positive attitudes Positive attitude 
helpful to researchers toward research toward research toward researchers 
517 .606 -0.721 -0.431 
Research Anxiety Scale 
p < .001* < .001* < .001* < .001* 
Note. N= 246. 
*p <.001. 


Effect of Research Anxiety on the Attitude Toward Scientific Research: Four simple linear regression models 
with Research Anxiety and subdimensions of Attitude Toward Scientific Research Scale were performed. The results 
showed that the students’ research anxiety affected their attitude toward scientific research. The first regression 
model explained 26.6% of the variance, the second 23.3%, the third 39.3%, and the fourth and final 9.3%. Table 5 
presents the findings. 


Table 5 
Effect of Students’ Research Anxiety on the Their Attitude Toward Scientific Research (n=246) 


Model 1 
Reluctance to be helpful to researchers 


Unstandardized Coefficients Standardized Coefficients 
t p 
B SE B 
Research Anxiety 387 042 516 9.258 <.001* 
Constant=7.982, F(1,245)=85.703, p < .001, R?= 26.6% 
Model 2 
Negative attitude toward research 
Unstandardized Coefficients Standardized Coefficients 
t p 
B SE B 
Research Anxiety 347 041 0.482 8.462 <.001* 


Constant=8,033, F(1,245)=71.614, p < .001, R?= 23.3% 
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Model 3 
Positive attitudes toward research 


Unstandardized Coefficients Standardized Coefficients 
t p 
B SE B 
Research Anxiety 0.444 036 0.627 -1.368 < 001" 
Constant=37.402, F(1,245)=152.976, p < .001, R? = 39.3% 
Model 4 
Positive attitude toward researchers 
Unstandardized Coefficients Standardized Coefficients ; 
B SE B 
Research Anxiety -0.206 042 -0.305 -4,926 < .001* 


Constant=30.298, F(1,245)=24.268, p < .001, R?=9.3% 


Note. N = 246. SE= standard errors 
*p <.001. 


Discussion 


Professionals must be inquisitive, learning-minded, inventive, creative, and self-reliant in order to retain their 
professionalism (Laaksonen et al., 2013). It is reported that midwives in the field do not show much interest in sci- 
entific research. The most important reason for this is that midwives do not have information about doing research 
(Guner et al., 2015). Undergraduate education is an important process in which students gain basic knowledge, 
application skills, and attitudes toward scientific research (Campasi & Finn, 2011; Gercek et al., 2016; Kes & Sahin, 
2019). Clinical practice is currently being guided by health sciences research and innovative scientific advancements 
(Al-Shalawy & Haleem, 2015). Students may have bad preconceived notions about research. Educating them on 
clinically accurate research results has the potential to modify their attitudes and increase their appreciation of 
research, perhaps enhancing midwifery practices. 

In studies conducted with nursing students, it was found that students’ anxiety toward conducting research 
was moderate (Kes & Sahin, 2019) or high (Gercek et al., 2016; GUleg & Saruhan, 2013). In most of these studies, 
more than half of the participants affirmed that conducting research made them nervous, anxious and scared 
(Amoo & Gbadamosi, 2021; Muzzmal et al., 2018). In this study, students got an average score of 23.92 + 9.29 on 
the anxiety scale. Given that the scale ranges from 12 to 60, we can assume that the degree of anxiety is quite low. 

As undergraduate education progresses, it is expected that students’ awareness of scientific research will 
increase and their attitude will change positively (Fernandez-Cano et al., 2021). There is mixed information about 
how the program level influences students’ views toward nursing research. Regarding some research results, 
fourth-year students had more positive attitudes toward research when compared with third-year students in the 
nursing education program (Halabi, 2016). Ina survey of third-year nursing students in Turkiye, they were found to 
have more positive views toward research (Unver et al., 2018). In another study conducted with nursing students, 
positive attitudes were found to be lower in the fourth grade (Kes & Sahin, 2019). In our study, students’ anxiety 
and attitudes toward scientific research do not change according to grade level (p > .05). Studies in the literature 
yielded similar results (ilhan et al., 2016). These results suggest that undergraduate education has a low effect on 
gaining a positive attitude toward research. Instructors should address the importance, benefits, and necessity of 
doing research frequently in educational activities. Educators should highlight the investigative role of midwives 
and encourage students to read scientific publications and conduct research. In this respect, they should provide 
them with the necessary consultation and support (Henricson et al., 2018). Students underline that the biggest 
impediment to undertaking research is a lack of mentoring and support (Arif et al., 2018; El Achi, 2020). One of 
the main functions of university education is to provide students with research experience and skills, as well as a 
positive attitude toward them. For this purpose, undergraduate education programs include courses on scientific 
research and statistics. 
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The anxiety level of the students who took the research methods course was significantly low in our study. 
Increased knowledge may have decreased anxiety. It was concluded that taking statistics or research methods 
courses did not affect the attitude toward scientific research. It is possible to see similar results in the literature 
(ilhan et al., 2016; Kes & Sahin, 2019; Yilmaz et al., 2020). In some studies conducted with nursing students, it was 
mentioned that students who participated in a scientific research course had more optimistic views (Halabi, 2016; 
Keib et al., 2017). The anxiety and attitudes toward research are expected to be positively influenced by the lessons 
on this topic (El Achi et al., 2020; Houlden et al., 2009; Ross & Burrell, 2019). The absence of the expected posi- 
tive effect can be interpreted as a low effectiveness of the courses. Reviewing these courses in terms of content, 
instructional approaches used, and practical exercises will be helpful. 

A significant proportion of students read and study papers relevant to their fields. Reading articles increases 
their knowledge and awareness of national and international professional developments and trends. It can also 
help to develop a positive attitude toward scientific research. As expected, it was observed that students who read 
articles had lower anxiety levels and had more positive attitudes toward researchers. A similar result was obtained 
in a study conducted with nursing students (Kes & Sahin, 2019). 

A total of 34.6% of the students follow periodicals in the field of health sciences. Following publications 
play an active role in professional development as well as increasing scientific knowledge sharing. In our study, 
students who follow periodicals in the field of health sciences have significantly lower anxiety levels and positive 
attitudes toward research. In the literature, it is seen that students who follow professional periodicals regularly 
have more positive attitudes toward research (Kes & Sahin, 2019; Unver et al., 2018; Yilmaz et al., 2020). Gaining 
the ability to follow scientific studies will help to raise the standard of care in professional life and to increase the 
use of evidence-based care practices (Fernandez-Cano et al., 2021). 

Of the students who took part in the study, the rate of students with experience in doing scientific research 
is 25.6%. This rate should be increased. Students should be encouraged and consulted. It was observed that 
students with scientific research experience had a significantly lower level of anxiety toward research, and their 
attitudes toward research and researchers were more positive. Similarly, students who want to do scientific re- 
search under the guidance of a counsellor have significantly lower anxiety levels and positive attitudes. Although 
the rate of students who have taken courses on statistics and scientific research and who have experience in 
doing scientific research is low, most of them want to do scientific research under the guidance of an advisor 
(67.9%). This rate indicates that there is a high motivation for students to do scientific research. This motiva- 
tion should be put to good use. Instructors should encourage pupils to do scientific research and provide the 
opportunity for them to do so. Only through practice can students begin to grasp the notion and significance 
of research (Tingen et al., 2009; Uysal-Toraman et al., 2017). There are findings in the literature that students 
participating in the scientific research process have more positive views toward doing research (El Achi et al., 
2020; Ross & Burrell, 2019; Unver et al., 2018; Uysal Toraman et al., 2017; Yilmaz et al., 2020). Having research 
experience contributes to students’ professional identity and improves their thinking and decision-making 
processes (Henttonen et al., 2021). Students’ research experience influences their perspective on research and 
enables them to gain positive views. The acquisition of research skills gives midwives the power and capacity 
to raise service standards. 

In the study, the rate of students attending scientific congresses/symposia is 33.7%. It was observed that 
students with this experience had significantly lower anxiety levels and positive attitudes toward research. The 
number of studies in the literature in line with this finding is quite large (Kes & Sahin, 2019; Unver et al., 2018). Some 
research results showed that participating in scientific activities did not affect anxiety levels and attitude toward 
scientific research (Ozdil et al., 2019; Yilmaz et al., 2020). To see the effect of scientific activities, it is necessary to 
increase the students’ awareness level as well as their participation in the activities. Students can acquire posi- 
tive attitudes by being supported in their involvement in scientific activities and encouraged to pursue scientific 
research. Teachers should prioritize enhancing undergraduate students’ exposure to medical research (El Achi et 
al., 2020). Finding a committed and enthusiastic mentor is a well-documented element that enhances student 
engagement in research, according to studies (DeVoe and Hess, 2018). 

In the study, it was observed that while the level of anxiety toward scientific research was not affected by 
increasing age, the attitude changed positively. As the age increases, the “Positive Attitude toward Research” in- 
creases and the “Reluctance to Help Researchers” decreases. There is conflicting evidence about the age’s effect on 
research attitudes (Halabi, 2016; Unver et al., 2018). Similarly, in a study conducted with nursing students, a positive 
correlation was found between the average age and the mean scores of the “Positive Attitude Toward Research” 
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sub-dimension (Unver et al., 2018). Halabi (2016) found that younger students had more positive attitudes. Another 
study found that older medical students had more optimistic views (Meraj et al., 2016). 

A significant relationship between research anxiety and all sub-dimensions of the attitude toward the research 
scale was found in the study. Increased anxiety in research enhances negative attitudes to scientific research and 
reduces positive attitudes. It is emphasized in the literature that a high level of research anxiety causes students 
to develop a negative attitude toward research (Gercek et al., 2016; Lei, 2008). Students cannot be expected to 
develop a positive attitude toward a situation that increases their anxiety level. Increasing knowledge and skills 
toward research will not only increase positive attitude but also decrease anxiety. 


Limitations 


Research participants were recruited from a single baccalaureate midwifery program in Turkiye. The results 
cannot be applied to all Turkish or international students studying midwifery. The study is restricted to the time it 
was conducted, the data collecting form created for the study, and the student replies. 


Conclusions and Implications 


As a result, midwifery students’ anxiety toward scientific research was found to be low and their attitudes 
toward scientific research were positive. Significant relationships were found between anxiety and attitude toward 
scientific research. Doing scientific research and attending scientific meetings decreases anxiety and positively 
affects attitudes. Students who have a negative attitude toward research may be reluctant and inefficient in evi- 
dence-based practices when they begin their careers. The acquisition of research knowledge and culture related 
to the profession begins with undergraduate education. For this reason, it should be ensured that during their 
undergraduate education, midwifery students cultivate a positive attitude toward research. In addition, developing 
a positive attitude will decrease research anxiety. It is clear that midwifery students are keen to conduct research, 
thus efforts should be made to develop their potential as researchers as soon as feasible. Undergraduate students 
should be actively involved in scientific research and activities. Opportunities should be created, and students 
should be supported. Early exposure to scientific research, contributes to building a strong midwifery education 
for students and quality midwifery care for women. 
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RESEARCH: A BIBLIOMETRICGC 
ANALYSIS 
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Ozlem Dugan, 
Bulent Cavas 


Introduction 


The dissemination of the research results of scientists and the presenta- 
tion of these results in a way that society can understand scientific informa- 
tion has led to the emergence of the concept of science communication. 
Especially while the increase in the number of mass media mediates the 
spread of scientific knowledge, the need for individuals to closely follow 
the developments in the field of science has made this issue important. It is 
a known fact that the high level of scientific literacy of individuals has been 
increased with the spread of science communication. 

As a result of the research conducted in the literature, the scarcity of 
bibliometric analyses in the field of science communication draws attention. 
It is predicted that the study carried out in order to reveal which studies have 
increased in the field of science communication and to determine the gap 
in this field will contribute to the field. 

In this context, the scarcity of bibliometric analysis in the field of science 
communication makes this research important and original. In the study, a 
detailed review of science communication literature was conducted, and a 
structured analysis was employed to explain the findings. In the study, 322 
articles were analyzed on science communication from the Web of Science 
database through the bibliometric analysis method, and their distribution 
was revealed by year, country, funding organization, research area, publishing 
house, country scores, and citations. 

In the field of science communication, the distribution of countries by 
years, which topics come to the fore, research topics, distribution of organi- 
zations supporting research, publishing houses, country scores and citation 
distributions are important in terms of revealing the position of science 
communication in the international research. It is also important to reveal in 
which countries science communication is studied more in the international 
research, country scores and country citation distributions in order to show 
which countries are at the forefront in the field of science communication. 
This study will also contribute to the academic field in terms of revealing how 
science communication has a visual mapping in the international research. 
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Abstract. /n recent years, the number of 
academic studies in the field of science 
communication has increased. It is 
important to make a general examination 
of the studies on science communication 
and to reveal their distribution according 
to years and countries in order to draw 

the framework of science communication 
studies. The main aim of this study was to 
analyze the science communication-based 
articles published in journals in the Web 
of Science (WoS) index in the last 22 years. 
Within the scope of the study, articles were 
scanned by typing keywords such as “topic”, 
“title” “keywords” science communication 
from the WoS database and 322 articles 
were examined by bibliometric analysis 
method. As a result of the study, the articles 
published between 2000 and 2022 were 
examined according to years, countries, 
funding organizations, research area, 
publishing houses, country scores and 
citations. According to the results, most 
articles were published in 2022(N = 58); 
USA, UK, Australia, Germany ranked first 
with the number of articles and SAGE (N = 
74) ranked first in publisher distributions. 
This study offers a global perspective on 
science communication and proposes a 
vision for future research. 
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Science Communication 


Science is a multidimensional concept with its social, cultural, democratic, and economic aspects (Burakgazi, 
2017). The multidimensional nature of science requires that scientific studies be made systematic by freeing 
them from dependence on communities and individuals (Gregory and Miller, 1998). The role of communication 
is important in the dissemination of science and its transmission to society. Burns et al. (2003), who made the 
definition of science communication, state that there is an uncertainty in the academic literature about science 
communication and related concepts. Science communication is defined as the use of information, reliable infor- 
mation between scientists, politics, society, deciders, industry, and other stakeholders (Burakgazi, 2017). Science 
communication reveals a dynamic structure for the production, dissemination of information, influence from social 
and cultural elements and interaction. Undoubtedly, science is moving away from its purpose to the extent that 
it is not transmitted; therefore, science communication is seen as one of the basic conditions for the existence of 
scientific knowledge. In this context, spreading science and increasing the level of scientific literacy depends on 
creating and maintaining science communication with strategic elements. Creating a qualified society and rising 
to the level of modern civilizations depends on increasing the level of scientific literacy (Utma, 2017). 

Although science communication is new as a concept in the field of communication, it dates back to the 19th 
century in terms of its field of study. Science communication has attracted the attention of researchers, politicians, 
and academics throughout history. Science communication also encourages individuals to participate in scientific 
discussions and policy. Science communication has become controversial, especially with the institutional, social, 
and technological changes and digitalization in the last few years (Babule et al., 2009). Rapid developments in 
Information and Communication technologies have made science communication more visible and traceable. 
While science communication has become widespread, especially through social media platforms, it can also be 
developed through various types of media, school education, seminars, museums and science centers, science 
clubs and competitions, various science events and science festivals (Burns et al., 2003). The sources of science 
communication are very diverse; newspapers and magazines, books, television, internet, science centers, museums, 
conferences, science festivals, etc. These communication tools contribute to establishing the relationship between 
scientists and scientific knowledge and society (Ozdemir & Kocek, 2020). Trench and Bucchi (2010) emphasized that 
science communication is developing in such a way that it intersects with social sciences, science education, mass 
communication, museum studies, academic and professional activities. If it is taken into account that scientists and 
society receive scientific developments from mass media (Hayes & Grossman, 2010), the importance of the media 
in terms of science communication becomes more obvious. Scientists are able to transmit their messages quickly 
and directly to different groups by using a wide variety of communication tools. 

Scientific knowledge is an important issue that determines the development level of countries (Tiryaki & 
Karakus, 2022). Although scientific and technological developments develop rapidly or slowly depending on the 
development of each country, developments in the field of science and technology are now much faster and it 
is becoming more difficult to track them. In this direction, the transfer of scientific knowledge is also becoming 
faster in the 21st century (Burakgazi, 2017). Science communication is seen as a personal, national, cultural, and 
political need (Lewenstein, 2014). As much as individuals in society need science, countries also need science 
communication for their development. Burns et al. (2003) defined the purpose of science communication as a 
process with scientific awareness, understanding and scientific culture. The increasingly central position of science 
and the discussions about healthy and sustainable living have made science communication important (Murcott 
& Williams, 2012). It is widely accepted that it is necessary to have a sufficient level of scientific literacy in order to 
be an effective member of society in a rapidly changing world (Ozdem et al., 2010). In this context, the concept 
of scientific media literacy has also come to the fore today. The way to increase scientific media literacy among 
the people is increasing in direct proportion to the studies carried out by the countries on this issue. Parameters 
of science communication, such as creating an attitude supporting science among the public and the participa- 
tion of the public in decision-making processes related to scientific and technological policies, can be possible 
with the establishment of an official science policy of a country and the existence of institutions that manage this 
process (Dursun, 2010). In this regard, every country should create, maintain and disseminate science policies to 
the extent possible. 

Science, which started to become a part of cultural life with public lectures and conferences, started to appear 
in newspapers for the first time in the 19th century (Dursun, 2010; Nelkin, 1987). In this context, the concept of 
science communication began to gain importance, especially in the 1980s, and efforts to communicate scientific 
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and technological developments to the public increased (Akoglu, 2011; Burns et al., 2003; Dursun, 2010). Studies 
on science policies as a country were first conducted by the USA. The science policy was established in the USA in 
1945 and started to be implemented (Elmaci, 2015). Science communication education is included in undergradu- 
ate, graduate and doctoral programs of universities in many countries such as China, India, Australia (Gascoigne & 
Metcalfe, 2017). Knight Science Journalism scholarship, awarded by Massachusetts Institute of Technology (MIT) 
in the USA, provides seminars and laboratory trips for journalists throughout the academic year to gain science 
writing skills and allows them to follow advanced science courses (Hayes & Grossman, 2010). Journalism education 
for scientists has been organized, so that scientists understand the perspectives of journalists and explain their 
workin a way that the public will understand. In 1969, the Exploratorium Science Center in the USA and the Ontario 
Science Center in Canada were established (Burakgazi, 2017). The American Science Development Association has 
been providing scholarship opportunities to graduate science and engineering students in media organizations 
since 1975. Founded in 1979, Science Communication journal started to publish theoretical and empirical research 
in the field of science communication (Burakgazi, 2017). Since 1987 in the UK, scientists have been sent to media 
organizations with journalism scholarships every year. In Turkey, science policy studies gained momentum with the 
establishment of TUBITAK in 1963. TUBITAK’s 2018-2022 Strategic Plan emphasises the importance of promoting a 
culture of science and technology. The establishment of the OECD in the development of science policies in Turkey 
has also increased the interest in scientific activities. The opening of 16 science centers in various cities in Turkey 
contributes to the dissemination of science to the public (Cakmakci & Gelmez-Burakgazi, 2020). 

Increasing undergraduate, graduate and doctoral level courses in the field of science communication is also 
important for the dissemination of science communication. Science courses should be made effective, especially 
in compulsory basic education, which is considered as one of the two most important tools in ensuring the sensi- 
tivity of society to scientific data. When evaluated from the point of view of science journalism, it is thought that 
the fact that basic science courses at the faculties of communication are elective for students. It is important for 
journalist candidates working in the field of science communication to have a level of understanding of English, 
to know how to access scientific information and how to convey the information they have reached to society ina 
simple and understandable language, even if they are not scientists (Kurtulus, 1997). It is also possible to state that 
book, paper and article studies have increased in the field of science communication. Academic studies aimed at 
measuring the level of knowledge of students related to science communication have also increased. In a study 
conducted in this area, in order to examine biotechnology students’ views on education in science communica- 
tion, 69 biotechnology undergraduate students were asked to rank the importance of 12 components (mass, 
language, content, context, interaction, style, prior knowledge, narrative, mode, dialogue, theory). A curriculum 
survey was conducted followed by semi-structured interviews with 13% of 2nd year students. As a result of the 
research, it has been revealed that students do not value science communication enough (Edmondston et al., 2010). 
Mercer-Mapstone and Kuchel (2015), in their research conducted on the teaching of science communication skills 
at the undergraduate level at 4 universities ranked internationally in Australia, found that more than 50% of 12 
basic communication skills are missing. In another study, a report prepared for the Australian Council of Deans of 
Science (ACDS) found that analytical, technical and problem-solving skills and science content knowledge were 
successfully taught, but communication skills lagged behind in oral, written and interpersonal communication 
types (McInnis et al., 2000). A UK-based study emphasized that expressing expectations when teaching communi- 
cation in a science graduate program is important for quality learning (Divan & Mason, 2015). The results of these 
studies reveal that science graduates should be equipped with communication skills. In the study conducted by 
AbersSek and AberSek (2010), it was stated that engineers need communication skills and should be trained in this 
regard. In this context, it has been determined that oratory and communication trainings are given to students 
studying in the engineering department in their research. As a result of the survey conducted after the training, 
it was determined that there was a remarkable improvement in the communication skills of the students (Cavas 
et al., 2013). Cavas et al., (2013) measured the scientific literacy level of science teacher candidates, revealed that 
the scientific literacy level of teacher candidates is high. Dagdan et al., (2015), in a qualitative study on pre-service 
teachers’ use of social media-supported scientific communication, determined that they could not perform the 
tasks of producing new and original ideas, comparing and presenting the emerging ones with previous studies at 
the stages that require a higher level of knowledge and skills. In the study, which examined 1300 articles published 
in the field of science communication between 1997 and 2018, it was revealed that science education gave more 
importance to communication methods to reach the public. In the same study, the increase in the interest of the 
media in science revealed the increase in media-themed research. Considering the role of the media in interacting 
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with the public, today the media is emerging as a basic component of science communication (Wu et al., 2019). 
According to the results of this study, it can be said that the media has an important role in the dissemination of 
science communication. 

Considering that the society’s need and curiosity for scientific knowledge have increased, it comes to the fore 
that scientists should also bridge the gap between society and science. Delivering the developments in the field of 
science and technology to ordinary people on the street is possible through science communication. In order for 
science to have a positive effect on society, there is a need for the scientific way of thinking to spread among the 
people. However, the further institutionalization of science in the 20th century has opened the distance between 
scientists and the public (Utma, 2017; Weingart & Guenther, 2016). Therefore, in order to close the distance be- 
tween society and scientists, communication experts, scientists and the media state that science communication 
should not operate as a one-way approach, as in the public’s understanding of science, but as a dialogue. In this 
field, since the 1990s, it has been stated that science should be discussed with the public and the public should 
understand science, and this approach has been called “public engagement with science” (Akoglu, 2010). Referring 
to the importance of dialogue in science communication, Kohen and Dori (2019) examined the articles published 
in 3 science communication journals and 3 science education journals published between 2000-2017. In the ar- 
ticles examined as a result of the research, it was determined that the importance of dialogue was emphasized in 
order to spread science communication. An important area where science educators and science communicators 
overlap is to increase the importance of science in society and involve various stakeholders in productive dialogues 
about science (Baram-Tsabari & Osborne, 2015). To enhance their communication effectiveness, scientists must 
comprehensively grasp and tactfully address the perspectives of interest groups, policymakers, businesses, and 
other key stakeholders involved in deliberations pertaining to decisions necessitating scientific expertise (Fischhoff 
& Scheufele, 2013). In this context, scientists should not hesitate to seek partnerships with talented communicators 
when communicating with the public (Bickford et al., 2012). An approach to dialogue-oriented public communica- 
tion requires scientists to be prepared to carefully consider the needs of their audiences and listen to their concerns 
(Trench et al., 2014). In addition, scientists should be careful when communicating with the media, and they should 
consider that the results of their research increase their responsibilities towards society. The main issue in the 
field is the existing distance between scientists and the public, which hinders effective science communication. 
Although current solutions highlight the importance of dialogue and public engagement with science, the main 
limitation lies in the lack of effective two-way communication. Consequently, there is a need to enhance science 
communication practices to ensure the broader dissemination of scientific knowledge and its positive impact on 
society. Addressing this need, the present research establishes a general framework based on academic articles 
published in the Web of Science database, within the context of science communication. 

This study includes data from 2000 to 2022, aiming to determine the subjects that have gained prominence 
in the field of science communication. By examining countries, funding institutions, research areas, publishing 
houses, country scores, and citations, valuable insights can be obtained regarding the evolution of the science 
communication field. 


Research Aim and Research Questions 


The aim of this study was to analyze the studies in the field of science communication and to reveal the trends 
in this subject. In this context, the research questions have been formed as follows: 

1. What is the distribution of articles indexed in the Web of Science database on science communication 
by year 

2. What is the distribution of the articles indexed in the Web of Science database on science communica- 
tion by country? 

3. Whatis the distribution of the articles indexed in the Web of Science database on science communica- 
tion by funding agencies? 

4. Howhas the distribution of articles indexed in the Web of Science database on science communication 
changed by research area? 

5. Whatis the distribution of the articles indexed in the Web of Science database on science communica- 
tion by publishers? 

6. Whatis the distribution of the articles indexed in the Web of Science database on science communica- 
tion according to country scores? 
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7. What is the distribution of the articles indexed in the Web of Science database on science communica- 
tion according to country citations? 

8. What is the distribution of the articles indexed in the Web of Science database on science communica- 
tion by keywords and what are the links of these keywords? 


Research Methodology 
General Background 


This study aimed to reveal the bibliometric analysis of the articles published between 2000 and 2022 on sci- 
ence communication. The study was conducted through a literature review and it was researched whether there 
are similar studies in the field of science communication. It was decided to conduct the study due to the scarcity 
of studies and the fact that the years 2000-2022 cover a wide period of time. It is known that the best method in 
such studies is the bibliometric analysis method. In this context, the articles obtained from the Web of Science 
database were analyzed by bibliometric analysis method. In the field of science communication, the distribution 
of countries by years, which topics come to the fore, research topics, distribution of organizations supporting re- 
search, publishing houses, country scores and citation distributions are important in terms of revealing the position 
of science communication in the international research. It is also important to reveal in which countries science 
communication is studied more in the international research, country scores and country citation distributions in 
order to show which countries are at the forefront in the field of science communication. 

Keywords related to science communication have been determined. It has been determined whether the 
keywords have changed over the years, and which topics and keywords have come to the fore. Within the scope 
of the study, the visuals of the prominent keywords in the field of science communication were also revealed 
with the VOSviewer programme. The distribution of keywords is also important in terms of showing which words 
are prominent in the field of science communication in the international research, determining which topics the 
academic community focuses on, and determining in which direction science communication has evolved. Within 
the framework of this approach, these determinations are important in terms of revealing the direction in which 
academic studies in the field of science communication have evolved. Bibliometric studies are effectively used to 
decipher trends and future perspectives between certain years. The bibliometric analysis method used to reveal 
the trends of the articles published in the Web of Science database in the field of science communication provides 
important data in terms of revealing the trends between years. 


Instrument and Procedures 


The Web of Science database has been used to determine the data related to science communication. Web 
of Science is a bibliometric database used to measure, evaluate and monitor scientific research. The database in 
question includes scientific articles and other studies published in scientific journals. WoS is a bibliometric database 
used to measure, evaluate and monitor scientific research. The database in question includes scientific articles 
and other studies published in scientific journals. It also includes research, methods and topics in various fields. 
With the Web of Science database, scientists can access article and journal evaluations such as the number of 
citations, impact factor and other criteria. It also includes research, methods and topics in various fields. With the 
WoS database, scientists can access article and journal evaluations such as the number of citations, impact factor 
and other criteria. In addition, scientists can discover new research areas by following publications and articles in 
a particular field (Scientific Publications, 2023). Bibliometric analysis can be employed to categorize aspects of sci- 
ence, such as the most contributing authors, journals, institutions, universities, and countries (Ellegaard & Wallin, 
2015). Bibliometric analysis was performed in the research. Bibliometry is defined as performing statistical analysis 
of publications related to a subject by using data such as year, country, source, subject, citation, and author (Cobo 
et al., 2011). Bibliometric analysis is the name of a general process required for science mapping, field analysis and 
visualization (Chen, 2017). The bibliometric analysis method provides the necessary information from analysis 
methods, data, networks and maps to create scientific maps. With this method, the densities in data, networks and 
maps can be determined. At the same time, it is possible to determine the degrees of strength of relationships, 
qualities and connections of data (Cobo et al., 2011; Jan Eck & Waltman, 2010). 
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The methodology developed and used by Tsai and Wen (2005) and Lee et al. (2009) was chosen to conduct the 
country analysis (Cavas, 2015). The accumulated score of each country was made according to the scoring made 
by Tsai and Wen (2005). In this form of scoring, “One point is given to each paper. If an article is published by more 
than one author, each from a different country, one point is divided into certain ratios for authorship from each 
participating country. For example, if an article is written by two authors, the first is a British author, the second 
is a US author, then the UK authorship for this particular article gets 0.6 points, while the US authorship gets 0.4 
points.” the expressed method has been used. 

The Web of Science (WoS-webofscience.com) database was used to access the published articles for bib- 
liometric analysis. Due to the reasonable availability of search filters, the keyword “Science Communication” has 
been used to find articles related to science communication in the search page of the database to access articles 
based on “Science Communication”. After searching on “Science Communication’, it was refined to get articles 
published from the year 2000. Also, editorial, mini-review, etc. articles are out of scope and this situation constitutes 
the limitation of the study. In addition, the criteria for including “Science Communication’ in the “title” “abstract 
or “keywords” of the studies were applied. The VOSviewer program was used to visualize the keywords. With the 
VOSviewer program, a bibliometric analysis of journals, analysis of specific subject areas, determination of word 
density in studies, analysis of website content and co-authorship can be done. The subject distributions of the 
journals, which vary according to the years, can be determined. Citation analyses of authors can be created. In 
the studies, the words that create density can be determined (Artsin, 2020). These determined elements can be 
revealed by visualizing with the VOSviewer program (McAllister et al., 2022). 


" 


Data Analysis 
In the study, 322 articles found as a result of filtering were analyzed by the bibliometric analysis method. A 
bibliometric analysis approach was used to provide explanations about the content of 322 articles based on Sci- 
ence Communication. The frequencies of the data obtained by the bibliometric analysis method were determined. 


Research Results 


The articles obtained from the Web of Science database were analyzed and the following results were obtained. 


Table 1 
Distribution by Years 
Years Number of Articles Years Number of Articles 

2022 58 2012 9 
2021 46 2011 5 
2020 49 2010 6 
2019 38 2009 3 
2018 18 2008 7 
2017 23 2007 3 
2016 23 2006 3 
2015 12 2005 1 
2014 11 2004 2 
2013 4 2000 1 
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When the distribution of the published articles by years is examined, 1 article was published in 2000 and 1 
article in 2005. It is seen that the studies in this field started to increase after 2014. Between 2000 and 2013, the 
number of article publications varied, decreasing in some years and increasing in others. It was determined that 
the same number of articles were published in 2016 (N = 23) and 2017 (N = 23). It was determined that more 
than 50 articles were published in 2022. It is possible to say that the number of articles increased every year in the 
period from 2000 to 2022. 


Table 2 
Distribution by Country 
Countries / Regions ead Bt Pounteles : Number of Countries / Regions Ruma of 
Article Regions Article Article 

USA 86 Portugal 9 Taiwan 2 
UK 62 Japan 8 Argentina 1 
Australia 36 Sweden ih Azerbaijan 1 
Germany 31 France 4 Botswana 1 
Canada 20 Israel 4 Colombia 1 
NewZealand 19 Mexico 4 Croatia 1 
Italy 15 Austria 3 Finland 1 
People’s Republic of China (PRC) 15 Russia 3 Greece 1 
South Africa 14 South Korea 3 Kenya 1 
Netherlands 13 Belgium 2 Lithuania 1 
Switzerland 13 Estonia 2 Malaysia 1 
Denmark 12 Belgium 2 Pakistan 1 
Spain 12 Kazakhstan 2 Thailand 1 
Brazil 10 Norway 2 Zimbabwe 1 
India 9 Singapore 2 


When the distribution of articles published by country is examined, it is seen that the USA (N= 86) comes to 
the fore. UK (N = 62) took the second place. Then there was a notable increase in Australia (N = 36) and Germany 
(N = 31). Although the number of article publications of Canada and New Zealand countries is close to each other, 
Canada (N = 20) and New Zealand (N = 19) have taken their places in the ranking with the article. 

The number of articles published in Italy and the People’s Republic of China is the same. It was determined 
that 15 articles were published in both countries. Likewise, it was determined that an equal number of articles (N= 
13) were published in The Netherlands and Switzerland. An equal number (N = 12) of articles have been published 
in Denmark and Spain. An equal proportion (N = 9) of the articles have been published in India and Portugal. It is 
observed that an equal number (N = 7) of articles are published in Ireland and Sweden. 

A total of 12 articles were published in France, Israel and Mexico in equal proportion (N = 4) in all three 
countries. A total of 9 articles were published in Austria, Russia and South Korea in equal proportion (N = 3), in all 
3 countries. A total of 14 articles were published in Belgium, Estonia, Indonesia, Kazakhstan, Norway Singapore, 
Taiwan in equal proportion (N = 2) and in these 7 countries. 

In the distribution of countries, it was determined that only one article was published in Argentina, Azerbaijan, 
Botswana, Colombia, Croatia, Finland, Greece, Kenya, Lithuania, Malaysia, Pakistan, Thailand, and Zimbabwe. The 
scarcity of articles published in the field of science communication in these countries is remarkable. 
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Table 3 


Country Distribution Scores 


2000 2004 2005 2006 
Country Score Country Score Country Score Country Score 
USA 1 Netherlands 2 Italy 1 Canada 1 
Italy 1 
South Korea 0.6 
UK 0.4 
2007 2008 2009 2010 
Country Score Country Score Country Score Country Score 
India 2 UK 2 India 1 Australia 1 
PRC i Belgium 1 Switzerland 1 Canada 1 
PRC 1 USA 1 Croatia 1 
USA 1 Germany 1 
Canada 0.68 Mexico 1 
Australia 0.64 USA 1 
Netherlands 0.47 
New Zealand 0.21 
2011 2012 2013 2014 
Country Score Country Score Country Score Country Score 
Denmark 1.6 India 2 UK 1 Australia a 
UK 1.4 USA 2 Italy 1 UK z 
Canada 1 PRC 1.17 Japan 1 USA Z 
USA 1 Canada 1 USA 1 Canada 14 
Finland 1 Italy 1 
Singapore 0.72 Spain 1 
Greece 0.53 Switzerland 0.6 
Germany 0.47 
Malaysia 0.11 
2015 2016 2017 2018 
Country Score Country Score Country Score Country Score 
UK 5.76 USA 3 USA 4.85 USA 4.34 
PRC 2 UK 3.4 Australia 4 Switzerland 2.32 
USA 2 Australia 3 UK 2.81 UK 1.6 
Russia 0.8 Denmark 2 Japan 2 South Korea (is 
Italy 0.79 Germany 1.93 South Africa 2 Brazil 1.47 
Kazakhstan 0.33 Brazil 1.61 New Zealand 1.19 Denmark 1.4 
NewZealand 0.32 Italy 1.6 India 1 New Zealand 1.4 
South Africa 1.47 Sweden 1 Japan 1.21 
France 1 Spain 1 Portugal 1 
Japan 1 Portugal 0.79 Australia 0.92 
New Zealand 1 Canada 0.68 Taiwan 0.6 
Switzerland 1 Indonesia 0.6 Canada 0.21 
Pakistan 0.6 Israel 0.47 
Mexico 0.26 Germany 0.4 
Argentina 0.13 Brazil 0.21 
2019 2020 2021 2022 
Country Score Country Score Country Score Country Score 
USA 8.21 USA 16.24 USA 11.07 USA 12.89 
New Zealand 5.01 Germany 6.82 Germany 5.16 Germany 7.53 
Australia a2 Australia 5.39 UK 4.98 Australia 5.07 
Germany 2.68 UK 3.72 Australia 4.18 Netherlands 4.08 
UK 2.23 Canada 2.66 Sweden 3 Spain 3.37 
Denmark 2 South Africa 1.07 Spain 2.74 PRC 3 
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Canada 1.64 France 1 Brazil 2 
Norway 1.4 Israel 1 Denmark 1.64 
South Africa 1.4 Italy 1 UK ilhoi 
Azerbaijan 1 Japan 1 Portugal 1.37 
Colombia 1 Lithuania 1 Switzerland 1.36 
India { New Zealand 1 New Zealand 1.06 
Israel { PRC 1 Canada 1 
Italy 1 Spain 1 India 1 
Taiwan { Sweden 1 France 0.6 
Switzerland 0.92 Switzerland 1 Indonesia 0.58 
PRC 0.92 Netherlands 1 Mexico 0.4 
Thailand 0.6 Zimbabwe 0.26 South Africa 0.4 
Spain 0.58 Botswana 0.24 Italy 0.26 
Brazil 0.53 Kenya 0.06 
Mexico 0.47 
Sweden 0.21 


In the last 7 years (between 2016 and 2022), the USA ranked first with 72.6 points in country score distributions. 
USA; increased its score, which was 1 in 2000, by 3 points in 2016, 4.85 points in 2017, 4.34 points in 2018, 8.21 
points in 2019, 16.24 points in 2020, 11.07 points in 2021, 12.89 points in 2022. The USA has generally increased 
its score each year. 

The UK is in second place with 31.81 points, Australia is in third place with 30.4 points and Germany is in 
fourth place with 25.99 points. Germany has surpassed other countries in the last 3 years (between 2020-2022). 
Germany scored 6.82 points in 2020, 5.16 points in 2021, and 7.53 points in 2022. Germany has increased its score 
in the last 3 years. 

Australia has increased its score over the last 4 years (2019-2022). The countries with the lowest scores are; 
Greece (0.53), Thailand (0.6), Pakistan (0.6), Taiwan (0.6), Argentina (0.13), Ireland (0.43), Indonesia (0.58) and Ka- 
zakhstan (0.33). In line with these numerical data, USA, UK, Australia and Germany stand out as the countries in 
the first place in terms of the number of articles published in the field of scientific communication. 


Figure 1 
Distribution of Citations by Country 
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According to the results of the citation analysis, the USA, Australia, the UK and Germany come to the fore. 
(Vosviewer Analysis). Figure 1 also includes the most cited country names. Figure 2 shows the results of the analysis 
of the countries where the authors are located. When Figure 2 is examined, the authors mainly; The USA, Australia, 
the UK and New Zealand countries come to the fore. Therefore, the results of the author and citation analyses are 
consistent. 

The USA received the most citations from Germany, Switzerland, and Italy. Germany; cited from Switzerland. 
Australia has received the most citations from the People’s Republic of China country. The UK, on the other hand, 
has received citations from both sides, in other words, from Ireland, Italy, the USA and the Netherlands, Australia. 

It can be stated that the countries that stand out in terms of the number of article publications also stand 
out in the number of citations. These countries are emerging as the USA, Australia, the UK and Germany. The high 
number of publications in the field of science communication brings about a high number of citations. 


Figure 2 
Country - Author Analysis 
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When the country-author analysis in Figure 2 is examined, the USA authors worked with Canadian authors, 
while the Australian authors worked with Canadian and New Zealand country writers. Country authors worked 
with Italy, New Zealand, Australia, Brazil, and Canada country authors. Among country authors, the USA, Australia 
and New Zealand show that country authors stand out and work solo. 

After the 2020s, country writers from the USA, UK, Italy, Canada, Brazil, New Zealand, and Australia come to 
the fore. 
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Table 4 
Publisher Distribution of Articles 
Publisher Names Sane Publisher Names re 
SAGE (USA) 74 American Society of Civil Engineers (ASCE) (USA) 1 
Acted sud Gian pludiAvanat 64 Channel View Publications (UK) 1 
Taylor & Francis (UK) 34 Duke University Press (USA) 1 
Frontiers Media Sa (Switzerland) 20 European Food Safety Authority (EFSA) (EU) 1 
Springer Nature (Germany) 20 Fuji Technology Press Ltd. (Switzerland) 1 
Wiley (USA) 13 Fundaco Oswaldo Cruz (Brazil) 1 
Elsevier (Holland) 11 Future Medicine Ltd. (UK) 1 
MDPI-Publisher of Open Access Journals (Switzerland) 8 Intellect Ltd. (UK) 1 
Indian Science Communication Society (ISCOS) (India 7 John Benjamins Publishing Company (Holland) 1 
Academia Brasileira de Ciencias (ABC) (Brazil) 4 Khazar University (Azerbaijan) 1 
Cogitatio Press (Portugal) 3 The Korean Earth Science Society (Korea) 1 
Emerald Group Publishing (UK) 3 Opragen Publications (UK) 1 
International Astronomical Union (IAU) (Japan) 3 Periodicum Biologorum (Croatia) 1 
Korean Academy of Medical Sciences (KAMS) (Korea) 3 Sci Methodical Ctr-Sci Educologica (Lithuania) 1 
Oxford University Press (UK) 3 Sciendo (Poland) 1 
Palgrave Macmillan (UK) 3 Social Neuroscience (USA) 1 
University of Valencia Botanical Garden (UV) (Spain) 3 Society for Technical Communication (STC) (USA) 1 
American Society for Microbiology (ASM) (USA) 2 Tecnoscienza (Italy) 1 
Amsterdam University Press (AUP) (Holland) 2 Tempus Publications (UK) 1 
Cambridge University Press (UK) 2 TripleC (UK) 1 
Canadian Journal of Communication (CJC) (Canada) 2 la ela oe Goa) eacher 1 
Canadian Science Publishing (CSP) (Canada) 2 Unisa Press (South Arica) 1 
Indian Academy of Sciences (India) 2 The University of Chicago Press (USA) 1 
Mary Ann Liebert, Inc. (USA) 2 University of Wisconsin-Extension Journal Inc. (USA) 1 
National Academy of Sciences (NAS) (USA) 2 University of the Orange Free State (South Africa) 1 
National Association of Biology Teachers, Inc. (USA) 2 University of Calabria (Italy) 1 
American Psychological Association (APA) (USA) 1 The White Horse Press (USA) 1 


When examined in terms of the publishers where the articles were published, it was found that the highest 
number of articles were published in the USA with 14 publishers. A total of 103 articles were published in USA 
publishing houses. With this number, the USA reveals that it has the highest number of publishers in the field of 
science communication. Italian publishers took the second place with 66 articles. 

It was determined that a total of 51 articles in the field of science communication were published by British 
publishers. A total of 29 articles were published by Swiss publishers, while 20 articles were published by German 
publishers. 
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It was determined that 14 articles were published by Dutch publishers, while 9 articles were published by 
Indian publishers. 5 articles were found worthy of publication by Brazilian publishers. It was determined that 4 
articles were published by Korean publishers. In other words, in the same issue, a total of 4 articles were published 
by Canadian publishers. 

3 articles were published in Portugal, 3 articles in Japan, 3 articles in Spain. It was determined that 2 articles 
were published by South African publishers. 1 article was published in Croatia, 1 article in Poland and 1 article in 
Azerbaijan. When evaluated in line with these results, it was determined that most articles were published by USA 
publishers. This country is followed by Italy, UK, Switzerland, Germany and the Netherlands. 


Table 5 
Refine by Research Areas (Categories) 

Research Areas er Research Areas vee 
Communication 177 Cell Biology 2 
History Philosophy of Science 62 Genetics Heredity 2 
Education Educational Research 39 Meteorology Atmospheric Sciences 2 
Environmental Sciences Ecology 25 Neurosciences Neurology 2 
Science Technology 25 Oceanography 2 
Social Sciences 16 Public Environmental Occupational Health 2 
Engineering 8 Sociology 2 
Psychology 8 Agriculture 1 
Philosophy i Anatomy Morphology 1 
Business Economics 5 Anthropology 1 
Information Science Library Science 5 Development Studies 1 
Life Sciences Biomedicine 5 Entomology 1 
Linguistics 5 Ethnic Studies 1 
Astronomy Astrophysics 4 Evolutionary Biology 1 
Computer Science 4 Food Science Technology 1 
Geology 4 Geography 1 
Area Studies 3 Government Law 1 
Biodiversity Conservation 3 Health Care Sciences Services 1 
Cultural Studies 3 Literature 1 
General Internal Medicine 3 Marine Freshwater Biology 1 
Plant Sciences 3 Pharmacology Pharmacy 1 
Social Issues 3 Public Administration 1 
Arts Humanities 2 Toxicology 1 
Asian Studies 2 Water Resources 1 
Biochemistry Molecular Biology 2 Zoology 1 


When examined in terms of the most preferred topics in the field of science communication, with 177 research 
areas, “communication” took the first place. This number was followed by the subject of “history philosophy of sci- 
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ence” with 62. It has been determined that 39 articles are related to the subject of “education research”. 25 articles 
were related to “environmental sciences ecology”. Agriculture, Anatomy Morphology, Anthropology, Development 
Studies, Entomology, Ethnic Studies, Evolutionary Biology, Food Science Technology, Geography, Government Law, 
Health Care Science Services, Literature, Marine Freshwater Biology, Pharmacology Pharmacy, Public Administra- 
tion, Toxicology, Water Resources, Zoology it was determined that the subjects were the least studied subjects. 


Table 6 
Distribution of Funds Supporting the Studies 
; ‘ Number of ; ' Number of 
Funding Agencies Article Funding Agencies Article 
National Science Foundation (NSF) (USA) 5 European Commission Horizon 2020 Research and D 
Innovation Funding Programme (EC) 
ae Fundacao De Amparo A Pesquisa Do Estado De Sao 
European Gommnlesion (EC) a Paulo Fapesp (FAPESP) (Brazil) 2 
UK Research and Innovation (UKRI) (UK) 5 Grants-in-Aid for Scientific Research (KAKENHI) 9 
(Japan) 
Economic and Social Research Council (ESRC) (UK) 4 Japan Society for the Promotion of Science (Japan) 2 
German Research Foundation (DFG) (Germany) 4 The Kavli Foundation (USA) 2 
Ministry of Education Culture Sports (Science And 3 Natural Sciences and Engineering Research Council of D 
Technology Japan (MEXT) (Japan) Canada (NSERC) (Canada) 
National Research Foundation (NRF) (South Africa) 3 National Science Foundation (NSF) (USA) 2 
; , Portuguese Foundation for Science and Technology 

Spanish Government (Spain) 3 (FCT) (Portugal) 2 
Australian Government (Australia) 2 Rita Allen Foundation (USA) 2 
CNPq-Conselho Nacional de Desenvolvimento Cienti- 9 Stiftung Mercator Schweiz (Switzerland) 9 
fico e Tecnologico-(Brazil) 
Danish Council for Independent Research- Humanities ; 
(FKK) (Denmark) 2 U.S. Department of Agriculture (USDA) (USA) 2 
Department of Tourism University of Otago New University of Otago Doctoral Scholarship (New 

2 2 
Zealand-(New Zealand) Zealand) 
Diane Campbell-Hunt Memorial Award-University of de it : : 
Otago (New Zealand) 2 The University of Seville (Spain) 2 
European Commission Joint Research Centre (JRC) 9 
(EC) 


In the study, the institutions that funded the articles were searched and as a result of the research, it was 
determined that a total of 70 articles received support. In the study, when the agencies providing funding support 
in terms of countries were examined, the USA funding agencies that supported 13 articles were in the first place. 

It was determined that EU funding agencies supported 9 articles. UK funding agencies supported 9 articles. 
While it was determined that the Japanese funding agencies supported 7 articles, it was determined that the New 
Zealand funding agencies supported 6 articles. While Spanish funding agencies supported 5 articles, German 
funding agencies supported 4 articles. 

Brazil's funding agencies supported 4 articles. South African funding agencies supported 3 articles. Denmark 
funding agencies supported 2 articles. Australian funding agencies appear to support 2 articles. 

Canadian funding agencies supported 2 articles. Portugal’s funding agencies supported 2 articles, while Swit- 
zerland's funding agencies supported 2 articles. As a result, it is seen that only 70 of the 322 articles examined in the 
study received support. When the funding agencies that support the articles are examined in terms of countries, 
USA, EU, UK, Japan and New Zealand countries are at the forefront. 
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In the study, the prominent keywords in the field of science communication were examined. A word cloud of 
keywords was created and the link between the keywords was examined. When the word cloud is examined, after 
2010, it is seen that the study subjects have become more prominent and intensified. 

It turns out that in the period from 2010 to 2020, the blue-colored subjects first switched to green and then 
yellow-colored subjects. In the focus of science communication, “taxonomy, “methodology”, quality’, “biotechnol- 
ogy education” blue-colored subjects have been forming decencies between 2010 and 2014. 


wu Wa, wu 


Green-colored topics are determined as “biotechnology”, “complexity”, “academic training’, “media training’, 


Wu, wu 


“training’, “science engagement’, “actor-network theory’, “curriculum”. The subjects in yellow are intensified in the 
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2020s. These issues come to the fore in the form of “public engagement’, “misformation’, “public communication 


of science’, “evaluation”, “mental models’, “boosting”. In recent years, it has emerged that studies on science com- 
munication and public, public and misinformation have increased. 


Discussion 


Societal needs for science and information are increasing day by day. Burakgazi (2017) defines science com- 
munication as the use of reliable information among scientists, policy, society, decision-makers, industry, and other 
stakeholders. While scientists share science and knowledge with the society, it is important that they are trained in 
communication. Science have fulfilled its purpose to the extent that it is shared with and delivered to the society. 
Gedrovics (2007), who stated that science is for people first of all, stated that interest in science is a driving force 
for the human mind. Science communication, above all, is seen as a personal, national, cultural, and political need 
(Lewenstein, 2014). Undoubtedly, science communication has become more visible to the society with the devel- 
opment of communication technologies. Considering that scientists and society receive scientific developments 
from the mass media (Hayes & Grossman, 2010), the importance of the media in terms of science communication 
becomes more evident. Using a wide variety of communication tools, scientists can quickly and directly convey their 
messages to different groups. However, it is important for scientists to receive training in science communication 
in order to present themselves and their scientific studies in a clearer and more understandable way. As a matter 
of fact, Abersek and Abersek (2010) state that engineers need communication skills and they should be trained in 
this regard. Edmondston et al., (2010) found that science communication was not given enough value in their study 
on students studying in the biotechnology department. In order to close the gap between science and society 
(Utma, 2017), more studies should be carried out in the field of science communication and the society should be 
exposed to scientific information (Kohen & Dori, 2019). In this context, articles on science communication scanned 
in the Web of Science database were examined in this study. English articles were found by typing “topic’, “title”, 
“keywords”. 322 articles were analyzed by bibliometric analysis method. Within the scope of the study, the articles 
published between 2000-2022 were examined according to years, countries, funds, research area, publishers, 
country scores and citations, and keywords. According to the results of the research, most articles were published 
in 2022 (N= 58). Among the countries, the USA stands out with the highest number of articles (N = 86). Then the 
UK, Australia, Germany took the first place with the number of articles. These results reveal that the USA is the first 
country to create science policies and the UK sends scientists to media organizations with journalism scholarships 
every year (Elmaci, 2015), and that they attach importance to the field of science communication and come to the 
fore in this field. It is also possible to evaluate this result in terms of publishers. In the study, it was revealed that 
the highest number of articles were published in the USA with 14 publishers. A total of 103 articles were published 
in USA publishers. The USA reveals that it has the most publishers in the field of science communication with this 
number. 66 articles have been published from Italy publishers and 51 articles have been published from UK pub- 
lishers. 29 articles were published in Switzerland, 20 articles in Germany, 14 articles in the Netherlands, 9 articles in 
India. Brazil, Korea, Canada, Portugal, Japan, Spain, South Africa, Croatia, Poland and Azerbaijan country publishers 
have published 5 or less than 5 articles. The USA, Italy and the UK are at the top of the publishers where the articles 
are published. In the last 7 years (between 2016 and 2022), the USA ranked first with 72.6 points in country score 
distributions. The USA has generally increased its score each year. These results reveal that the USA has come to 
the forefront with its studies in the field of science communication. The UK is in second place with 31.81 points, 
Australia is in the third place with 30.4 points and Germany is in the fourth place with 25.99 points. Germany has 
surpassed other countries in the last 3 years (between 2020-2022). Germany scored 6.82 points in 2020, 5.16 points 
in 2021, and 7.53 points in 2022. Germany increased its score in 2022. Australia has increased its score over the last 
A years (2019-2022). The countries with the lowest scores are; Greece (0.53), Wales (0.6), Thailand (0.6), Pakistan (0.6), 
Taiwan (0.6), Argentina (0.13), Ireland (0.43), Indonesia (0.58), Kazakhstan (0.33). In line with these numeric data, 
the USA, Australia, the UK and Germany stand out as the countries that take place in the first place. 

The USA in terms of citation, received the most citations from Germany, Switzerland, and Italy. Germany; Cited 
from Switzerland. Australia; People’s Republic of China has the most citations from China. UK; cited from both sides, 
namely Ireland, Italy, USA and Netherlands, Australia. When the country-author analysis was examined, the USA 
country writer worked with Canadian authors, while the Australian countries authors worked with Canadian and 
New Zealand country writers. New Zealand country writers worked with the UK and Italy country writers. The UK 
country authors worked with Italy, New Zealand, Australia and Canada country authors. Among country authors, 
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the USA, Australia and New Zealand show that country authors stand out and work solo. After the 2020s, country 
writers from the USA, the UK, Italy, Canada, Brazil, New Zealand and Australia come to the fore. In terms of the num- 
ber of articles, the USA and Australia come to the fore. These numerical data reveal that the results are consistent. 

When the study is evaluated in terms of research areas; The issue of “communication” is more prominent. 177 
research areas have been related to the subject of “communication” This number was followed by the subject of his- 
tory philosophy of science with 62. 39 articles are about education research. 25 articles are related to environmental 
sciences ecology. For example, studies in the field of climate change science communication are increasing. 1175 
articles were obtained and analyzed in the WoS database on this subject (Chen et al., 2022). In the research conducted 
in the Scopus database with the keywords environment and communication, it was determined that there was an 
increase in environmental communication studies (Akerlof et al., 2022). A bibliometric analysis of the articles published 
on the Green Economy was conducted (Albayrak, 2023). A study was also carried out on green jobs (Durmaz et al., 
2023). In the bibliometric analysis of publications obtained from SCI and SSCI databases between 1998 and 2019, it 
has been determined that industrial wastewater is among the most studied topics in recent years (Mao et al., 2021). 
Agriculture, Anatomy Morphology, Anthropology, Development Studies, Entomology, Ethnic Studies, Evolutionary 
Biology, Food Science Technology, Geography, Government Law, Health Care Science Services, Literature, Marine 
Freshwater Biology, Pharmacology Pharmacy, Public Administration, Toxicology, Water Resources, Zoology were the 
least studied subjects. It is predicted that the increase in the articles published on these subjects will also ensure 
the spread of science communication. With the continuous development of communication technology, studies in 
the field of communication have started to come to the fore. For example, 1,157 articles on science-based virtual 
brand communities between 2000 and 2020 were revealed by bibliometric analysis (Zheng et al., 2023). In the field 
of sports crisis communication, 1,283 scientific publications were examined by bibliometric analysis (Harker & Saffer, 
2018). Ozcinar (2021) conducted a bibliometric analysis of communication education between 1990-2020. Between 
1975 and 2016, when many articles on human resources education were published, 900 articles were reached and 
analyzed with bibliometric analysis management (Danvila-del-Valle et al., 2019). In another study conducted with 
data obtained from the Web of Science database between 2013 and 2022, it was found that social sciences are inter- 
disciplinary, public administration, environmental studies, urban studies, psychology are multidisciplinary, followed 
by educational research, communication, management, political science and law (Catone, 2023). 

In the study, the institutions that funded the articles were searched and as a result of the research, it was de- 
termined that a total of 70 articles received support. In the study, when the institutions providing funding support 
in terms of countries were examined, the USA funding agencies that supported 13 articles were in the first place. It 
was determined that EU funders supported 9 articles and UK funders supported 9 articles. It was determined that 
Japanese funding agencies supported 7 articles. It was determined that New Zealand funders supported 6 articles. 
Spanish funding organizations supported 5 articles. Germany funders supported 4 articles and Brazil funders sup- 
ported 4 articles. South African funders supported 3 articles. Denmark, Australia, Canada, Portugal and Switzerland 
funding agencies supported 2 articles, a total of 5 countries supported 10 articles. As a result, it is seen that only 
70 of the 322 articles examined in the study received support. When the funding organizations that support the 
articles are examined in terms of countries, USA, EU, UK, Japan and New Zealand countries are at the forefront. 

Within the scope of the study, keywords were determined in the field of science communication. A word cloud 
of keywords has been created. When the word cloud is examined, it is seen that the study topics have become 
more specific and intensified after 2010. It turns out that from 2010 to 2020, the blue-colored subjects first shifted 
to green, then yellow-colored subjects (Figure 3). In the focus of science communication, blue-colored topics such 


Wu wn, ou 


as “taxonomy”, “methodology”, “quality”, “biotechnology education” are intensified between 2010-2014. Green- 
colored topics are determined as “biotechnology”, “complexity”, “academic training’, “media training’, “training’, 
“science engagement’, “actor-network theory’, “curriculum”. The subjects in yellow are intensified in the 2020s. These 
issues come to the fore in the form of “public engagement’, “misformation’, “public communication of science’, 


“evaluation’, “mental models’, “boosting”. In recent years, it has emerged that studies on science communication 
and public, public and misinformation have increased. In the research conducted by creating the “communication” 
category in the journals indexed in the Web of Science Journal Citation Report, the topics of internet, public rela- 
tions, advertising, health, relationship, discourse, news, telecommunications, public opinion, activism came to the 
fore between 1980-2013 (Montero Diaz et al., 2018). De las Heras-Pedrosa et al., (2022), who revealed the mapping 
analysis of the COVID-19 pandemic process, found that the concepts of vaccination, infodemi, risk perception, social 
distance and telemedicine were more involved in academic studies in 2021. In the global bibliometric analysis of 


the COVID-19 pandemic, the USA and China came to the fore in the academic field (Wang &Tian, 2021). Kuipers et 
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al., (2022), who examined the articles in crisis and disaster journals between January 1, 2020, and June 30, 2022, 
with bibliometric analysis, it was determined that the concepts of risk, crisis communication, governance, mental 
health, resilience, and vulnerability came to the fore. In the bibliometric analysis on 21 st century skills in the field 
of education, it was determined that the concepts of technology, higher education, cooperation, critical think- 
ing, creativity, pedagogy, education, teacher education, and evaluation were more prominent between the years 
2000-2022 (Akcan et al., 2023). In the research conducted from the WoS database between 2015 and 2020, 12,272 
publications on e-learning were reached. In the study, it was determined that the USA, Spain and China came to 
the fore (Djeki et al., 2022). In the field of digitalization and tourism (Ozogul Balyali & Akgis Ilhan, 2023), political 
communication and social media issues come to the fore (Saf, 2023). 

In line with the results of the study, it is revealed that most studies in the field of communication sciences 
were carried out by the USA. Considering the number of articles published by the countries, their country scores, 
publishers and institutions providing funding, it can be stated that the countries with the lowest scores in the 
ranking and the countries that are not included in the ranking should also pay attention to the issue of science 
communication and work on this issue. In the study conducted by Kohen and Dori (2019), articles published in 
3 science communication journals and 3 science education journals published between 2000 and 2017 were ex- 
amined, and it was determined that the importance of dialogue was emphasized in the articles in the examined 
journals as a result of the research. In this regard, Baram-Tsabari and Osborne (2015) emphasize that the common 
point of science educators and science communicators is to increase the importance of science in society and to 
involve various stakeholders in productive dialogues about science. 


Conclusions and Implications 


This study examines articles in the Web of Science database within the international literature on science 
communication from 2000 to 2022, using specific parameters. The study reveals the evolution of studies in the field, 
presenting the distribution of articles across years, countries, funding organizations, research areas, publication 
companies, country scores, and citations. The study provides important results and contributions for researchers 
in science communication by offering insights into paradigms in the field and a global perspective. Furthermore, 
it proposes a vision for future research. 

The findings indicate that science communication is a developing field in science education and serves as a 
strong evaluation tool for science education policymakers and researchers, in addition to providing bibliometric 
indications. Consequently, science communication should be acknowledged as a study topic and receive sufficient 
financial support for extensive research, along with inclusion in organizational work programs. Benchmarking and 
ongoing monitoring of science communication levels are necessary to assess the success of projects and programs 
for all societal actors, not limited to students. 

In conclusion, this study provides a comprehensive view of science communication research as a whole. By 
analyzing data from scientific papers, it demonstrates the changes in the field from 2000 to 2022. Based on these 
findings, the need for fostering coordinated and interdisciplinary collaboration between the scientific community 
and society is emphasized, leading to stronger and more reliable partnerships. The study anticipates that its results 
will assist experts and decision-makers in understanding the current state of science communication research and 
provide guidance for future endeavors. 
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TEACHER’S GAZE BLIND SPOT 
IN SCIENCE LECTURE CLASS 


Jung-Ho Byeon, 
Yong-Ju Kwon 


Introduction 


In a life that requires social interaction, human beings constantly per- 
form interactions called social communication with others and respond ap- 
propriately by analyzing and processing external information or signals. The 
fact that humans can perceive and deal with numerous pieces of information 
through the eyes among various sensory organs means social interaction 
with others is very important for survival. Humans continuously deal with 
external information through intentional and unintentional attention at every 
moment, and their attention could increase when they give an interested 
or detailed observation of an object or event. In particular, eye contact and 
watching face is a powerful visual cue to build social connections between 
communicating persons and affects various cognitive processes of commu- 
nication (Madipakkam et al., 2015; Schilbach, 2015). In addition, looking at 
the other person is a primary behavior to send intention for communication 
because it is an essential means of non-verbal communication and also af- 
fects verbal communication (Jiang et al., 2017). 

The teacher should confront a heavy load for external information 
processing during class, and they must continuously interact with many stu- 
dents as long as the class. From a pedagogical perspective, social interaction 
between teachers and students should emphasize for efficient learning, and 
teachers need to provide interaction and attention evenly to a large number 
of students. Actually, can teachers pay equal attention to students in actual 
classroom situations? 


Attention 


Visual attention and cognitive processing of information occur almost 
simultaneously, and measurement results of attention provide information 
about the subject's intentions and objectives in a specific situation (Beach 
& McConnel, 2019; Tatler & Land, 2015). In particular, human attention takes 
place by the interaction between endogenous and exogenous attention 
factors, and exogenous attention could be induced by signals or information 
encouraging visual attention to be continuously interpreted by following gaze 
and repeatedly affect (Tatlet & Land, 2015; Veneri et al., 2012). According to 
the theory of mind, humans respond more efficiently in processing the other 
person's facial information than detectible other information. Humans have 
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Abstract. The teacher's gaze attention can 
trigger interaction with the student. So if 
the teacher fails to equal attention during 
class, students may be alienated from the 
interaction. According to this perspective, 
this study aimed to establish the pattern 

of the teacher's gaze during science 

lecture classes and whether the change 

of the gaze when the student’s seat as an 
external factor changed. Eye tracking was 
conducted on six teachers during class, and 
the teacher's gaze fixation and movement 
were also analyzed after changing the 
student seat. According to the results, 
teachers mainly focused on the center of 
student seats, and the gaze blind spot was 
mainly biased forward. Even if the student’s 
seat was changed, the gaze was focused 
on the center of the classroom, and the 
gaze blind spot differs within the individual, 
and the teacher tends to be unaware of it 
himself. Consequently, the teacher's gaze 
concentration pattern is generally similar, 
but the gaze blind spot varies depending on 
the teacher and can be affected by external 
factors. Therefore, it is necessary to develop 
a system and retrain program for diagnosis 
and feedback of teachers’ attention to 
provide proper and high-quality education 
to students. 
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a cognitive tendency called social attention to predict and recognize the other person's feelings or thoughts by 
interpreting various information such as the gaze and facial expressions of others. Social attention is a tendency 
and interpretative process to focus one’s gaze on the eyes of others to understand their cognitive and emotional 
state (Emery, 2000; Jiang et al., 2017; Senju & Johnson, 2009). 

The teacher's attention-related teaching behavior in class can explain in the context of social attention. Social 
attention can be explained through developmental psychological elements such as mutual gaze, following gaze, 
joint attention, shared attention, and theory of mind (Emery, 2000; Jiang et al., 2017; Senju & Johnson, 2009). In 
social attention, eye contact results from an interaction between teacher and student caused by the teacher's at- 
tention toward the student as instruction behavior (Emery, 2000; Jiang et al., 2017; Senju & Johnson, 2009). The 
cognitive psychological process from eye contact to the theory of mind is to infer the psychological state of the 
other person by judging extrinsic response signals such as facial expressions and reactions when eye contact 
between students and teachers occurs. 


Teaching Competency and Attention 


Teachers’teaching skill in class is a factor of the teachers’ professional competency, which individually controls 
the learning of students participating in the class and is essential in student development (Baumenrt & Kunter, 
2013). In general, the teacher professionally expresses the quality of teaching in class, and it can be divided into 
the efficiency of class management, supportive class atmosphere, and cognitive encouragement (Kliem et al., 2009; 
Pianta & Hamre, 2009). Positive interactions between teachers and students are the main component of a sup- 
portive atmosphere and are known to be related to the development of student's intrinsic motivation and interest 
from the perspective of sociocultural constructivism (Fauth et al., 2019; Kunter et al., 2013; Lazarides et al., 2018). 

Teachers’ professional competence can reveal through teaching behavior and teacher-student interaction, and 
it can mediate by teachers’ interactive behavior during class (Mahler et al., 2017). According to a recent study, the 
interaction between teacher and student during class affects student interest and satisfaction with their learning 
(Fauth et al., 2019). Because teachers decide which information and signals to pay attention to, the learner's par- 
ticipation is determined by the teacher's behavior and response during class (Wolff et al., 2016). Therefore, during 
class, teachers should constantly find clues in students’ behavior about their participation and whether they focus 
on learning (Goldberg et al., 2021). Teachers’ professional competence is an essential functions instruction factor 
in all subject classes. Particularly science teachers should ask for not only concept learning but also the additional 
operation of scientific inquiry that includes experimental activity (Furtak et al., 2012). However, conducting scien- 
tific inquiry in class requires teachers to have high science literacy and challenging performance. For this reason, 
teachers generally practice lecture-style teaching focused on concept learning more than inquiry learning. Teach- 
ers’ teaching style is mainly lecture type due to a lack of experience or confidence in professional knowledge for 
inquiry learning (Brobst et al., 2017; Johnston & Ahtee, 2006). 

The social interaction between teachers and students, including the professional competency of the teacher, 
should be set as the main factor of their class to provide students with meaningful learning in a science lecture class. 
The relationship between teacher and student can connect through teaching behavior and learning behavior at 
every moment in a class, and it can be represented through social attention by gaze fixation (Pennings et al., 2018). 
From this perspective, measuring teachers’ teaching behavior during class is necessary to analyze teachers’ eye 
movements, class interactions, and students’ satisfaction with teachers’ professional development and feedback 
(Cortina et al., 2015). It could measure the difference in attention tendency as each teacher by eye tracking, and it 
could suitably analyze teachers’ interactive attention during class (King et al., 2019; Wolff et al., 2016). 


Eye-tracking 


In the context of cognition, eye tracking can provide quantitative data on the cognitive process that regulates 
human behavior so that it can provide objective information about teachers’ visual perception and attention during 
class (Shayan et al., 2017). Because instruction is composed through complex interactions between teachers and 
students in real-time, studies on the attention of teachers have mainly focused on specific activities such as language 
reading, math problem-solving, and multimedia instruction (Haataja et al., 2019; Hunt et al., 2015; Norqvist et al., 
2019; Rayner 2009). Some studies present analysis results on attention focused in class according to the teacher's 
career (van den Bogert et al., 2014; McIntyre et al., 2017; Wolff et al., 2016). 
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The distribution of teacher's gazes to students is uneven because teachers communicate through gaze, and 
social interactions during class repeatedly affect teacher's visual attention (van den Bogert et al., 2014; Dessus et 
al., 2016; McIntyre et al., 2017; Prieto et al., 2017). In addition, experienced teachers mainly use top-down percep- 
tual mechanisms, and there is an attention tendency to enhance students’ learning by consciously focusing on 
information relevant to the lesson (Wolff et al., 2016). Therefore, the result of measurement on social interaction 
between students and teachers can be able to use quantitative data for comparison of individual differences by 
the analysis of where the teacher focuses his attention during class. Not only does the analysis result of where the 
gaze is direct but also the fixation time of the gaze reflects the teacher’s cognitive process during class. In related 
studies, professional teachers pay attention to students for a long time compared to less experienced teachers, 
and a short fixation time on students means that their eyes move to other objects rather than interaction or their 
attention is dispersed (van der Boget et al., 2014; Mclyntyre et al., 2017). On the other hand, the increased complex- 
ity and student number make it more difficult for teachers to manage classes and cause relatively short fixation 
times for students (Prieto et al., 2017). 

The visual attention of the teacher can be measured using the number of times the gaze fixation, the dura- 
tion of the gaze fixation, and the consistency of the pattern during the learning process, which can be used as 
an efficient scale to explain the cognitive process (Haataja et al., 2019; Hunt et al., 2015). Gaze fixation provides 
information about what the teacher is paying attention to during the class, and the consistency pattern shows the 
teacher's tendency to focus on methods. Teachers encourage students’ learning participation by communicating 
through eye contact and inducing learning-related behaviors during class (Dessus et al., 2016; Korthagen et al., 
2014). Therefore, in this study, we will investigate the teacher’s gaze concentration and consistency during class 
to understand what the teacher’s gaze focuses on and is distributed during science class. 


Research Hypotheses 


Research on teachers visual attention during classes mainly investigates the differences in attention patterns 
between expert and novice teachers (Dougusoy-Taylan & Cagiltay, 2014; Korthagen et al., 2014). However, in many 
research, the number of teachers who participated in eye-tracking is limited to 1 to 4, so it tends to perform quanti- 
tative analysis to complement the difficulty of generalization the findings (Dagiené et al., 2021). The measurement 
data of teachers’ attention during class enables quantitative analysis of teaching behavior because it can provide 
microscopic and sequential information on interactive teaching behavior (Magnussen et al., 2017). Therefore, two 
hypotheses were set to investigate the teacher's attention pattern and the student's perception of the class in an 
actual science lecture class situation. In general, science inquiry learning requires behavioral interactions such 
as student individual activity guidance, experiment performance assistance, and feedback. However, in science 
lecture classes where behavioral interactions are minimized, the primary interaction method between teachers 
and students may be limited to visual attention and speaking. 

For this reason, teachers should focus on students as evenly as possible by using their professional class 
management capabilities during class. However, humans have individually non-intentional endogenous attention 
tendencies and intentional exogenous attention strategies (Fernandez et al., 2021; Ogmen et al., 2016). Therefore, 
if the intervention of unintentional endogenous attention mainly affects to teacher's attention, the teacher could 
be challenged to pay attention to all students evenly during class. On the other hand, if the intentional attention 
strategy is mainly a teaching behavior, they can pay attention equally to all students. Therefore, researchers estab- 
lished the first hypothesis to confirm as follows. 


Hypothesis I: There are differences in teachers’ gaze attention to students in science lecture classes. 


Exogenous attention is a response triggered by a signal or cue that arouses attention and corresponds to an 
intentional action to acquire information (Goldberg et al., 2021). Teachers process and respond to various informa- 
tion and signals discovered from students in instruction situations in class. The generated reactions immediately 
act as a feedback factor that induces attention and can affect the teacher’s attention. In this context, students with 
higher levels of academic achievement perform more actively in class than students who do not, and students with 
high-level confidence in their abilities participate in more discussions (Boheim et al., 2020; Goldberg et al., 2021). 
In general, one of the student factors that induce a teacher's attention is the existence of students with excellent 
academic ability or students who attract attention, whether positive or negative (Stahnke & Blomeke, 2021). If 
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the presence or absence of a specific student corresponding to the extrinsic attention factor affects the teacher's 
attention pattern, the teacher's attention will lateralize depending on where the student is present. Conversely, if 
whether or not an outstanding student does not affect the teacher's attention pattern, the teacher's attention will 
appear constant regardless of the presence or absence of an interested student. Therefore, researchers established 
the second hypothesis to confirm as follows. 


Hypothesis II: The presence or absence of students who arouse gaze attention affects the attention pat- 
tern of teachers. 


Research Methodology 
General Background 


Intrinsic tendencies and external stimuli induce the human gaze, and the relationship between teachers and 
students is formed by interactive processes that occur every moment in class (Pennings et al., 2018; Tatler & Land, 
2015).In addition, social interaction with students in class affects teachers’ gaze attention, and individual teachers’ 
teaching behavior could differ depending on relationship orientation (Dessus et al., 2016; McIntyre et al., 2017; 
Prieto et al., 2017). This study set up two hypotheses to clarify the effect of endogenous and exogenous variables 
on teachers’ gaze attention distribution as teaching behavior. Mainly, it is necessary to control the student vari- 
able and investigate only the characteristics caused by the teacher variable to clarify the relationship between 
the teacher's gaze distribution and teaching behavior in science lecture classes. For this condition, analyzing the 
teacher-specific gaze concentration areas and gaze distribution tendencies measured several times when different 
science teachers teach in the same classroom with the same students is essential. While most schools and teachers 
do not agree to continue recording classes with an eye tracker, six science teachers working at school A in South 
Korea participated in this study. The researcher received the school’s official permission and the voluntary research 
participation of 6 teachers and 63 students. Therefore, two classes in that school were recruited as teaching places 
to investigate teachers’ gaze attention. 


Participants 


All six science teachers graduated from a university providing preservice teacher training courses in Korea 
and possessed teacher licenses certified by the state (Table 1). All participants underwent a gaze test and had 
uncorrected visual acuity of 1.0 or better or corrected visual acuity of 1.0 or better. This study was conducted in 
compliance with the declaration of Helsinki and followed a protocol approved by the Ethics Committee of KNUE, 
and all participants gave their written informed consent. All participants had normal or corrected to normal visual 
acuity but had no history of psychiatric disorder and Strabismus. Also, participants made sure of right-handedness 
according to the Edinburgh handedness inventory. 


Table 1 
Participants 
Class(N) Teacher Major of university Age Education career 
Al Physics education 34 5 years 
A (30) A2 Chemical education 27 0.5 years 
A3 Biology education 30 2 years 
B4 Chemical education 29 3 years 
B (33) B5 Earth science education 57 30 years 
B6 Biology education 36 10 years 
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The students were divided into A and B groups through their teaching class name and school curriculum to 
maintain regular learning courses in school. The eye tracking was performed on six high school science teachers 
to investigate the teacher's attention and movement patterns during science class. The eye movement data were 
acquired by mobile eye tracker during each science class and analyzed gaze fixation and movement on all AOls. The 
AOI set for three higher AOls and the thirteen sub-AOls (nine student areas, three teaching and learning material 
areas, and a non-teaching area) to investigate whether or not the blind spots in science lectures. 

Teachers wore the eye tracker until they felt no discomfort from wearing the eye tracker during class. Teachers 
who participated in this study could intentionally adjust gaze distribution due to the self-recognition that they 
are tracking eye movement, and students can also show unusual interactions by expressing curiosity and interest 
in teachers wearing eye trackers. A double-blind test was conducted on all teachers and students to offset this 
interference and test the first hypothesis. In two classrooms, three teachers each wore eye trackers in the same 
classroom and conducted several lessons, and both teachers and students answered that the eye trackers caused 
no curiosity effect or discomfort, and then the eye tracking data was used for analysis. The researcher randomly 
changed the student seat to verify the second hypothesis between the class used by measuring the actual gaze 
tracking data and the next class and measurement. The teacher was unaware of the unexpected change of stu- 
dent seats in advance, and the students’ positions in the student area divided into nine were randomly adjusted 
by researchers so that they did not match before and after the seat change. The second gaze data collection was 
conducted during the next class without the teacher’s prior recognition of how the student's position changed 
for the blind test. 


Eye-tracking 


The eye movements of subjects were recorded using a goggle-type eye tracker (ViewPoint Eye Tracker®, Ar- 
rinton Research, Inc.) to collect the teacher’s attention data during class. The eye tracker consists of ahead camera 
(56°, horizontal FOV) that captures the scene where the subject is looking and an infrared camera that records 
the position of the pupil and cornea. The head position was monitored through the display screen of the tracker 
body and checked whether or not is tracking the pupil. In addition, the spatial resolution of the eye tracker is 0.15 
°, the tracking signal generation unit is 60 Hz, and the accuracy is 0.25 °- 1.0 °. According to the dominant eye test, 
the eye-mount camera position changed before the eye movement data collection. The foveal coordination was 
obtained through pupil tracking by infrared projection extracted in two-dimensional plane coordinates. A calibra- 
tion to compensate for eye torsion was performed every time before recording each subject. The calibration was 
performed according to nine points while the teachers’ heads stabilized to eliminate measurement errors by posi- 
tion distortion that could occur driven by individual eye differences. After eye tracking, all participants reported 
their actual thinking of each task through an in-depth interview. 


Data Analysis 


The data analysis was performed through the Viewpoint program, and the area of interest (AOI) was set up 
and defined for analyzing fixation and saccade. The areas of interest for eye movement analysis were set as student 
area, teaching and learning material area, and non-teaching area. The student area is where students are sitting 
from the teacher’s point of view, and the teaching and learning material area means the teacher's focus on teach- 
ing activities, such as textbooks, TV monitors connected to computers, and blackboards. Also, the areas unrelated 
to teaching and learning, such as walls and windows, were divided into non-teaching areas, and gaze movements 
were analyzed. After the definition of AOI, post-process was performed on individual eye movement data to extract 
data such as frequency of attention, gaze movement, and gaze path. The gaze fixation threshold was set at 200 ms 
to conduct the analysis and to minimize intentional attention to the AOI. 

The coordinate matrix is constructed based on the scene template, and the two-dimensional coordinate is 
calculated from gaze data extracted eye camera using MatLab R2016b (9.1). After data computation, gaze data 
synchronized to the scene template. According to the purpose of this study, in the next step, gaze fixation was 
defined as the amount of continuous time (200 milliseconds) spent looking within an AOI. The total time spent fixa- 
tion of the main task was calculated as the sum of fixations within each AOls. The total number of gaze fixations for 
each AOl is defined as the total number of gaze fixations, and the gaze fixation ratio was calculated by calculating 
the number of gaze fixations of lower AOls for all AOls. Fixation ratio means fixed gaze number rate of total gaze 
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fixation number within AOI. The time-series information of moving to another AOI after fixing the gaze on an AOI 
was extracted and used for cross-analysis on the gaze path. In addition, gaze movement analysis utilized data ex- 
tracted from the degree of gaze movement to another AOI immediately after gaze fixation on one AOI for 200 ms. 
The teacher's gaze movement pattern was confirmed within the student area to analyze what strategies implied 
in the teacher's teaching behavior that distributes attention during class. After gaze fixation and movement occurred 
in one of the nine student areas was divided into four attention types ‘induced attention by behavior;‘exploratory 
attention for knowledge,’exploratory attention for interaction, and ‘social attention for interaction’ depending on 
where the subsequent eye fixation occurs. Induced attention by behavior is a case where the subsequent gaze 
fixation moves to the non-teaching area immediately after the gaze fixation occurs in the teaching and learning 
material or the student area. It occurs when the teacher moves, or the attention is unrelated to teaching and learning. 
Exploratory attention for knowledge was determined as a movement to the teaching and learning material area 
after gaze fixation occurs in the non-teaching or student areas, and continuous gaze fixation occurs in the same 
area within the teaching and learning material area. Exploratory attention for interaction determined to move to 
a different student area from the first one after gaze fixation occurred in one of the student areas. Social attention 
for interaction means subsequent fixation in the same place after gaze fixation occurred in the student area. The 
analysis of gaze movement and attention interpreted how the teacher's teaching behavior strategy changed by 
comparing changes before and after replacing students’ seats. Since the number of gaze fixations obtained from 
12 times gaze trace from 6 people differed, it needs to relative rate value for teacher comparison. The relative ratio 
was calculated as a percentage based on the total number of gaze fixations shown in individual measurements. 


Research Results 
Verification of Hypothesis | 


Researchers used eye trackers to collect eye movement data from 6 science teachers in two classes. Eye 
tracking was performed during a science lecture class to analyze differences in attention distribution as teaching 
behavior for social communication with students. In addition, after eye-tracking measurement, the difference be- 
tween the attention distribution pattern shown in the eye-tracking result and the attention pattern recognized by 
the individual teachers was compared using the survey response results on their teaching and learning behavior 
tendencies individual teachers thought (Table 2). 


Table 2 
Gaze Fixation Rate of Teachers on AOIs during Lecture Class 


Student area Teaching and learning area Non-teaching area 
Teacher - 6 FS 
% % % 
Al 39.59 56.97 3.44 
A2 28.24 36.61 35.16 
A3 49.68 36.94 13.38 
B4 40.12 53.09 6.79 
B5 43.04 47.17 9.78 
B6 42.36 48.00 9.65 


In the case of class A, some teachers paid more attention to the comparison area, others paid more attention 
to the student area, and the rest of the teachers paid the most attention to the teaching and learning material area. 
In the case of class B, most of the teachers’ eyes were mainly distributed in the teaching and learning material area, 


418 


Ass https://doi.org/10.33225/jbse/23.22.413 


Journal of Baltic Science Education, Vol. 22, No. 3, 2023 


ISSN 1648-3898 /Print/ TEACHER’S GAZE BLIND SPOT IN SCIENCE LECTURE GLASS 
(PP. 413-426) 


ISSN 2538-7138 /oniine 


but much attention was also happening in the student area. Therefore, it can be seen that there are individual dif- 
ferences in the distribution of eyes in the student area, teaching and learning material area, and non-teaching area. 
In addition, it was confirmed that there is a tendency for more attention to be paid to the teaching and learning 
material area than the student area. Table 3 shows the results of dividing the student area into three horizontal 
and vertical sections from the teacher's point of view to confirm the tendency of attention to the student area. 


Table 3 
Average Rate of Gaze Fixation of Teachers on Student Area 


ae LF Lc LR CF cc CR RF RC RR 
% % % % % % % % % 

Al 0.65" 2.65 3.81 2.01 10.45" 9.64 1.61 4.04 4.73 
A2 1.93 2.15 2.53 6.22 6.23" 3.21 0.86" 2.80 2.30 
A3 1.61 1.52 6.45 2.67 14.52" 9.89 1.27" 7.04 471 
B4 1.78 3.24 2.65 3.89 8.28 10.55" 1.41" 3.93 4.40 
BS 1.37" 2.93 3.17 8.06 15.23" 3.69 2.25 3.90 2.45 
B6 2.05 2.85 1.49" 9.63 10.15" 4.33 4.82 4.74 2.30 


Note. The gaze fixation ratio of nine AOls on the student area was calculated, and areas with maximum and minimum values 
were indicated within the student area (LF: Left-Front, LC: Left-Center, LR: Left-Rear, CF: Center-Front, CC: Center-Center, CR: 
Center-Rear, RF: Right-Front, RC: Right-Center, and RR: Right-Rear). 

™ Maximum value of average fixation rate on student area. 

™ Minimum value of average fixation rate on student area. 


On a horizontal basis, the teacher's gaze on the left, center, and right student areas mainly focused on the 
center. On the other hand, on a vertical basis, there were individual differences among teachers and differences 
according to the number of measurements for the front, center, and rear areas. These results mean that teachers’ 
gazes during class have similar tendencies regarding horizontal lines, and differences may exist depending on in- 
dividual classes regarding vertical lines. Since the human gaze moves around the linkage point when focusing on 
an object (Gesierich et al., 2008), the most attention can appear in the center based on the horizontal line. On the 
other hand, the differences in the vertical line standard can be attributed to differences in the internal disposition 
of individual teachers or differences in external signals by class. 

After completing the gaze tracking, it compared to the self-report and gaze tracking results where they did 
not usually focus their attention during science classes. Six teachers answered that the eyes were relatively not 
focused on the left side based on the horizontal line and were relatively not distributed to the front for A1, A2, and 
B6 teachers and A3, B4, and B5 teachers. In this study, the area of interest where the teacher's attention is relatively 
low during class will call as the term ‘blind spot’ Comparing the blind spots identified through eye tracking and 
the blind spots presented by the teacher through the survey, the other five, except teacher A, did not match the 
actual eye concentration results and self-report results (Table 4). This result means that in terms of teaching and 
learning, gaze control, which is purposeful behavior, may not be accurately recognized by the teacher. In other 
words, the teacher may not accurately remember how achieved the adjustment of gaze concentration is related 
to social interaction during class. Therefore, since it is challenging to self-discover the bias of gaze control that 
appeared in the class, it is necessary to receive professional measurement and feedback to discover and improve 
the problem of teaching behavior. 
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Table 4 
Result of Gaze Tracking and Survey about Blind Spots on Student Area 


Blind spot of horizontal standard Blind spot of vertical standard 
Peaches —§_ 
Gaze tracking Survey Match Gaze tracking Survey Match 
Al Left Left ° Front Front fo) 
A2 Right Left x Rear Front x 
A3 Left Left ° Front Rear x 
B4 Left Left ° Front Rear x 
BS Left Left ° Rear Rear ) 
B6 Left Left ° Rear Front x 


Note. 0 = match between gaze tracking and survey , x = mismatch between gaze tracking and survey. 


The above results showed individual differences and general tendency in the opening of the teacher's attention 
pattern on the student area during the lecture class presented in the first hypothesis. In addition, it was confirmed 
that blind spots that occur in the actual focus of attention and blind spots that teachers perceive themselves might 
be different. Therefore, professional consulting and feedback from a teaching behavior perspective would need 
rather than self-diagnosis to strengthen the interaction between student and teacher. 


Verification of Hypothesis II 


The second hypothesis was verified by checking whether there was a change in the gaze concentration pat- 
tern before and after changing the seat where students were sitting. If a teacher focuses relatively more attention 
on students with outstanding academic achievement or response, the pattern of attention before and after the 
change of student seat will be different because the external attention signal strongly affects them. On the con- 
trary, if the gaze concentration before and after the change of seat is the same or similar, the internal attention 
tendency is stronger than the external attention signal provided to the teacher. Table 5 shows the minimum and 
maximum gaze rate of the nine student areas in the total gaze concentration of the class before and after replac- 
ing the student position. 


Table 5 
Gaze Fixation Rate of Teachers within Student Area 


Teacher Time LF Lc LR CF cc CR RF RC RR 
% % % % % % % % % 

Pre 2.05m 742 748 4.89 19.32 28.88 M 5.28 13.78 10.91 

‘ Post 1.05m 5.65 12.71 5.37 36.61M 17.83 2.32 5.02 13.44 
Pre 8.99 7.54 6.62 24.88M 17.98 13.68 2.05m 9.95 8.31 

- Post 4.53 7.70 11.53 18.93 26.53M 8.82 4.10m 9.91 7.96 
Pre 5.02 2.88 3.23 6.16 50.12M 15.49 1.94m 10.18 4.99 

ne Post 1.20m 3.28 24.16 4.48 5.29 24.97 M 3.28 18.73 14.61 
Pre 6.71 8.71 4.86 7.07 19.62 28.70 M 3.72m 10.40 10.21 

Post 2.34m 748 8.22 12.09 21.56 24.07 M 3.34 9.23 11.68 
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Teacher Time LF Lc LR CF cc CR RF RC RR 
% % % % % % % % % 

Pre 4.89 2.72 6.29 20.38 41.83M 3.80 1.68m 11.30 7.10 

ie Post 01.53m 10.76 8.42 17.10 29.10M 13.22 8.68 6.88 4.31 

Pre 08.03 12.47 6.41 9.41 26.39M 17.55 3.35m 9.18 7.22 

= Post 01.63 0.90 0.60 m 36.22M 21.52 2.83 19.49 13.20 3.62 


Note. The gaze fixation ratio of nine AOls on the student area was calculated, and areas with maximum and minimum values 
were indicated within the student area (LF: Left-Front, LC: Left-Center, LR: Left-Rear, CF: Center-Front, CC: Center-Center, CR: 
Center-Rear, RF: Right-Front, RC: Right-Center, and RR: Right-Rear). 

M Maximum value of average fixation rate on student area. 

™ Minimum value of average fixation rate on student area. 


In teachers A1, A2, A3, and B6, the highest attention area slightly changed before and after the seat change of 
students, but there was a tendency to focus on the center based on the horizontal line. Teachers B4 and B5 focused 
the most on the same area before and after the change of student seats and focused on the center based on the 
horizontal line. On the other hand, there were differences between individuals and classes in the concentration of 
attention based on the vertical line within the center area. 

As a result of confirming the blind spot within the student area, teacher A1 had the lowest fixation ratio in 
the LF area, and A2 had the lowest value in the RF area. Teachers A3, B4, and B5 showed the lowest gaze fixation 
on the LF and RF area, but teacher B6 was the lowest value on the LR and RF area. Since teachers A1 and A2 had 
the lowest fixation rate in the same area in all classes, they seemed to be affected by the intrinsic factor of the 
teacher having a significant influence on the formation of blind spots rather than the influence of external signals. 
On the other hand, Teacher A3, B4, and B5 are partially affected by changes in external factors but tend to be bi- 
ased toward the front area. However, teacher B6 seems to have a pattern of attention distribution dependent on 
changes in external factors like the change of student seat. This result is because there is an individual difference 
in the tendency of gaze concentration to form blind spots in a specific area based on the horizontal or vertical axis. 
However, most areas with relatively large gaze distribution were the center areas, with no significant individual 
difference. Therefore, an individual’s tendency to concentrate may differ between internal and external factors. In 
addition, the blind spot for the location of the student area from the teacher's point of view is mainly formed on 
the left and right sides compared to the center based on the horizontal line. Therefore, during class, the blind spot 
of the horizontal standard is formed on the left or right side due to the influence of internal and external factors 
and mainly provides relatively more attention to the center of the classroom. 

It was confirmed whether the individual patterns of where the teachers who participated in the study fo- 
cused their attention during the class and in what order they focused their attention were consistent and similar. 
Cross-analysis was conducted by extracting time series data on where attention was sequentially focused before 
and after student position replacement from the teacher's attention data obtained during science lecture classes. 
As a result of performing the chi-square test on the gaze path for each teacher before and after the student seat 
replacement, a significant difference was not found in most teachers but found in one teacher (Table 6). 

Teacher A1 (x2 = 149.923, df = 132, p = .136), A3 (y2 = 106.318, df = 132, p = .951), B4 (y? = 131.411, df= 121, 
p = .244), BS (2 = 137.191, df = 120, p = .135) and B6 (x2 = 151.723, df = 132, p = .115), no statistical difference 
was found in the gaze path before and after the student seat change, but teacher A2 (y? = 155.986, df= 121, p= 
.018) showed differences at a statistically significant level. In other words, five teachers have no difference in gaze 
path, but there is a difference in 1teacher. These results mean that most teachers have a similarity and a specific 
regular pattern in the order of gaze distribution in classes. Therefore, to confirm the attention strategy implied in 
the gaze distribution as a teaching action, four types of gaze movement patterns were classified by analyzing the 
case where the gaze continuously moved from one specific area of interest to another. The movement of attention 
to the non-teaching area due to the teacher's behavior or head movement was defined as induced attention by 
behavior. The gaze movement at the teaching and learning material area or stayed continuously was defined as 
exploration attention for knowledge. Also, when the gaze moves from one AOI to another within the student area, 
it is defined as exploratory attention for interaction, and continuous attention in the same area within the student 
area is defined as social attention for interaction. 
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Teachers A1, B4, B5, and B6 showed the most exploratory attention for knowledge occurred, and social at- 
tention for interaction was the second significant gaze movement. However, the second rank in B5 differed in the 
pre-post measurement, but both were related to the interaction. In the case of teacher A2, induced attention by 
behavior and exploratory attention for knowledge were crossed in the first and second places in each class. Except 
for A2, five teachers mainly showed exploratory attention for knowledge and social attention for interaction in 
all classes (Table 6). Therefore, it could interpret that a specific knowledge transfer-oriented lecture class is being 
conducted, with the majority of attention related to teaching and learning materials to check the contents of sci- 
ence knowledge and present them to students. In addition, researchers found that the teacher constantly monitors 
whether the contents are delivered according to the goal by implementing interactive attention to check whether 
the students understand the science contents during the knowledge delivery lecture. 

On the other hand, in the case of A2 teachers, there was a different type of attention pattern from other 
teachers, but the teacher had less than one year of experience in classroom instruction than other teachers. In 
particular, as a beginner teacher, it shows a distracting type of attention that shows a relatively large amount of 
close attention caused by other behaviors in class compared to interactive activities. 


Table 6 
Attention Type of Science Teachers during Science Lecture Class 


Teacher Time IAB EAK EAI SAI 
% % % % 

Pre 5.09 48.11 19.22 27.57 

. Post 1.78 65.84 12.96 19.42 
Pre 39.41 31.17 10.57 18.85 

. Post 30.89 42.05 10.58 16.47 
Pre 15.48 31.46 16.12 36.94 

i: Post 11.28 42.44 12.76 33.53 
Pre 3.24 58.32 15.43 23.01 

e Post 10.35 47.86 17.10 24.70 
Pre 16.08 41.94 36.53 5.45 

e Post 3.76 52.59 12.83 30.82 
Pre 11.22 46.19 11.49 31.10 

i Post 8.07 49.82 14.90 27.21 


Note. IAB: induced attention by behavior, EAK: exploratory attention for knowledge, EAI: exploratory attention for interaction, 
SAI: social attention for interaction. 


From the teacher's point of view, the distribution of attention to students was concentrated in the center 
based on the horizontal line, and the attention area before and after differed by individual and class in vertical 
standard. In addition, except for one case, 11 blind spots focused on gaze were also found to be biased in the front 
area. In other words, the teacher's attention during class was mainly at the center position, and the blind spot was 
mainly at the front. Therefore, it can be said that the individual difference in the location of the blind spot on the 
horizontal line or the concentration area on the vertical line can partially affect the distribution of attention by 
exogenous factors related to some student or class. In addition, there was no statistical difference in the analysis 
of the gaze path before and after the student seat replacement, and the gaze movement pattern was consistent 
with the other experienced teachers except for the case of one novice teacher. Therefore, the teacher's focus area 
during science lecture classes does not change much even if the student’s seat changes, but the concentration 
and blindness based on the horizontal line can be affected. 
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Discussion 


Development for students in the educational context is possible when realizing universal primary and sec- 
ondary education, so it is necessary to ensure that all students receive equal and quality education effectively in 
classroom instruction. Quality education requires various goals and measures, and the development and support 
of teachers’ professional teaching competencies are particularly essential topics. Providing an adequate diagnosis 
and feedback system for appropriate education and professional development for teachers can positively affect 
teachers who significantly influence students’learning outcomes (Fauth et al., 2019; Kleickman et al., 2016; Lazarides 
etal., 2018). In this context, gaze distribution related to interaction skills with students during class is a professional 
competency held by teachers, and appropriate diagnosis and feedback are essential for effective student learn- 
ing outcomes. However, the focus and distribution of teachers’ attention during actual classes are challenging to 
diagnose and provide feedback, and there are few opportunities for professional advice or education. 

Eye contact, the starting point of social attention, triggers teacher and student interaction and corresponds 
to teaching behavior essential to all classes, including lecture-style and activity-oriented classes (Goldberg et al., 
2021; Pennings et al., 2018). For accurate diagnosis and feedback for teachers, measurement results suitable for 
individuals should be provided based on quantitative data, so this study challenged using a gaze tracking method. 
Some case studies have been presented on how teachers’ gaze in class, known as common sense, takes place in 
the actual class, but most studies focus on what context novice and expert teachers concentrate on while watch- 
ing class videos (van den Bogert et al., 2014; McIntyre et al., 2017; Wolff et al., 2016). In this study, we analyzed the 
concentration and movement of teachers in the process of actual classes for students and whether the teacher’s 
gaze distribution pattern changed before and after students changed their seats in the same class. 

We investigated the attentional patterns of teachers during science lecture classes, where linguistic expres- 
sions play a leading role in contrast to active learning activities such as inquiry-based learning. The result of gaze 
concentration analysis revealed individual differences or group-level general tendencies within the gaze data into 
student, teaching and learning material, and non-teaching areas. These differences could be related to variations 
in the degree of reliance on teaching and learning materials among teachers during the class. In particular, most 
teachers showed high levels of gaze concentration on the teaching and learning material area, which can be ex- 
plained by the characteristic of lecture classes that continuously confirm and review scientific concepts conveyed 
to students (Leuchter et al., 2014). Creating a positive atmosphere and maintaining cognitive activity is essential 
for teachers conducting classes as part of their teaching abilities (Fauth et al., 2014). Therefore, teachers make 
continuous efforts to monitor their students and verify the content of the course during class. As a result, teachers 
would generally focus more on teaching and learning materials and students during the class. 

On the other hand, the level of gaze concentration in the student area showed significant individual differ- 
ences along with a general tendency. Humans do not jump instantly to a distant object when switching their gaze 
from one object to another. Instead, there is a tendency to focus on the existing object during the gaze shift and 
use it as an intermediate point to shift attention to the next object (Gesierich et al., 2008). Therefore, regardless 
of whether teachers move their gaze to the left or right of the vertical line, they tend to focus on the students in 
the center as the midpoint. In contrast, individual teacher differences were evident for the front, middle, and rear 
positions based on the vertical line. Moreover, vertical line-based differences were observed differently depend- 
ing on the class, even for the same teacher. The correlation analysis was performed to determine whether these 
differences were related to age or educational experience, but statistical significance with age or educational 
experience was not identified. 

In conclusion, the concentration of gaze on a particular student area during class is mainly due to humans 
endogenous attentional bias towards the center. On the other hand, blind spots in gaze are more dependent on 
individual differences and mainly affected by changes in student seating, especially along the vertical axis. Therefore, 
when conducting science lecture classes, the position where teachers and students can interact most frequently 
through social attention is in the center of the classroom, and front seats on both sides are likely to be excluded 
from interaction with teachers. Furthermore, while most teachers know where their gaze is primarily focused along 
the horizontal axis, only a few properly recognize their focus along the vertical axis. 
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Conclusions and Implications 


To achieve equal and high-quality education and assistance for students in the field of education through 
classroom instruction, teachers have to possess teaching and learning competencies that allow for personalized 
interaction with students and management of the classroom. However, this research reveals variations in teachers’ 
gaze distribution tendency during science lecture classes, despite the general similarity of teachers. Teachers’ gaze 
concentration was mainly focused on the center of the classroom, and gaze blind spots occurred mainly at the front 
left and right sides. Therefore, students are relatively excluded from social interaction opportunities with teachers 
during classes, and recognition and improvement by self is a challenging problem for teachers. Consequently, the 
concentration of gaze on a particular student area during class is mainly due to humans’ endogenous attentional 
bias towards the center. 

On the other hand, blind spots in gaze are more dependent on individual differences and mainly affected by 
changes in student seating, especially along the vertical axis. Therefore, when conducting science lecture classes, 
the position where teachers and students can interact most frequently through social attention is in the center of 
the classroom, and front seats on both sides are likely to be excluded from interaction with teachers. Furthermore, 
while most teachers know where their gaze is primarily focused along the horizontal axis, only a few properly rec- 
ognize their focus along the vertical axis. As a result, in science lecture classes, it is difficult for teachers to distribute 
their attention to students regarding social interaction evenly. It is challenging to expect voluntary correction as 
they may not be aware of the possibility of exclusion. Therefore, there is a need to provide a feedback system that 
can diagnose and correct teaching behaviors related to teaching competency for interaction with the student. Just 
as gaze tracking simulation and feedback systems are utilized to maintain pilots’ operation skills at a high level, a 
solution system and program for enhancing teaching behaviors need to propose. In addition, efforts must establish 
the scientific infrastructure and education program for developing and training related to teaching behaviors in 
teacher training and re-education institutions. 
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GAMIFIED EDUCATIONAL 
ROBOTS LEAD AN INCREASE IN 
MOTIVATION AND CREATIVITY IN 
STEM EDUCATION 


Tan-I Chen, 
Shih-Kai Lin, 
Hung-Chang Chung 


Introduction 


With the development of time and the progress of technology, educa- 
tional reforms are being carried out in various countries to promote com- 
petitiveness and demand for STEM education. The alignment of different 
courses in the field and the curriculum design enables students to continue 
and integrate learning in various areas and interdisciplinary learning. The 
current curriculum designs emphasize students’ problem-solving ability, deep 
learning, multidisciplinary knowledge integration, and practical application. 
STEM, the interdisciplinary teaching model combining science, technology, 
engineering, and mathematics, provides students with many skills needed 
in the 21st century, such as complicated communication, problem-solving, 
and teamwork (Lei et al., 2020). The learning content of the STEM curriculum 
provides students with specific subject knowledge that connects them to the 
ability to integrate interdisciplinary knowledge and further promote learning 
outcomes (Sailer & Sailer, 2021). The STEM curriculum design also emphasizes 
real-world situations or actual problems as a backdrop for students to solve 
problems using exploratory and problem-based methods. 

The knowledge covered in the STEM curriculum is vast and complicated, 
and the content could be more abstract and quickly learned. Previous re- 
search has shown that learning motivation is essential to students’ learning 
performance (Bai et al., 2020). Unfamiliar or uninteresting content would 
eventually lead to low student learning motivation. A gamification curriculum 
could effectively arouse students’ learning interest. The term gamification cur- 
riculum refers to integrating elements or mechanisms of video games, such as 
scoreboards, quizzes, and badges, into the classroom. Previous research has 
found that such a method can improve learning focus, motivation, problem- 
solving skills, and student willingness to participate (Sanchez et al., 2020). 
Chevalier et al. (2020) analyzed the results of an educational robot activity 
for problem-solving. Their results showed that the training helped enhance 
students’ cognitive processes related to problem understanding, idea gen- 
eration, and solution formulation, helped them develop a well-established 
strategy for approaching learning robot problems, and could be an effective 
method for scaffolding. 
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Abstract. The combination of education 
and robots is becoming an increasingly 
important issue. Although researchers have 
conducted some studies on educational 
robotics in STEM education at the 
undergraduate level, no research examined 
the effects of gamified educational 
robotics on participants’ motivation and 
creativity. Therefore, this study examines 
the effects of gamified educational robots 
on participants’ learning motivation and 
creativity in STEM. A quasi-experimental 
research design was used in this study. 

A total of 108 students from two classes 
enrolled in the information technology 
course were involved as participants. 
Learning motivation and creativity 

scales were used to collect data. The 
participants were introduced to a gamified 
educational robot as the course objective, 
learning content, and game mechanism 

to increase motivation and help students 
solve problems. In contrast, students in 

the control group received conventional 
instruction consisting of group discussions 
and lectures. Activities in both groups 

were held for eighteen weeks in three 
phases. The results showed that gamified 
educational robots could enhance learning 
motivation and positively influence 
learners’ creativity. The results also indicate 
that learning motivation has significant 
effects on creativity, and students with 
high motivation perform better in terms 

of creativity. The results strengthen 
interdisciplinary STEM teaching and 
promote students’ learning outcomes. 
Keywords: gamification in STEM 
education, educational robot, STEM 
curriculum, learning motivation 
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In a recent systematic review of studies on educational robotics, Anwar et al. (2019) found 53 studies on 
educational robotics that addressed creativity and student motivation. The studies in their analyses considered 
motivational aspects of creativity to improve students’ motivation and interest in STEM courses and program- 
ming. They found that educational robotics was used in the studies to encourage and increase students’ interest in 
STEM concepts. They also indicated that engaging students in learning robotics could increase student creativity. 
To parallel this result, Gubenko et al. (2021) have emphasized that gamification can support the implementation 
of educational robotics interventions. Sailer et al. (2013) have stated that gamification and game elements could 
stimulate motivational mechanisms and foster motivation. The results from this research indicate that gamified 
learning robots have the potential to attract and increase students’ learning motivation and interest. It was found 
that they used gamified educational robots in the classroom and introduced them to support students’ learning 
process in programming to improve students’ engagement in learning the content of the STEM curriculum (Madar- 
iaga et al., 2023). Robotics education could motivate students to study STEM-related subjects (Daher, 2022) and 
positively support the national demand for capacity in the STEM fields. 

A gamified robotics teaching could integrate STEM education to attract students, help students identify and 
solve problems with multidisciplinary knowledge, and help students overcome the career barrier in STEM. Research 
has shown that using educational robots in STEM curricula positively influences students’ problem-solving and 
teamwork skills (Chevalier et al., 2020; Madariaga et al., 2023). Some research has found that gamified educational 
robots can improve students’ engagement with the STEM curriculum, attitude, problem-solving, and learning 
motivation (Reyes et al., 2021). In a newly published study, Leonard et al. (2016) indicated that using robotics and 
game design encouraged some students to design creative and interesting games. 

Gamification involves applying game elements in different situations (Sanchez et al., 2020). Game elements 
are designed to achieve specific goals and precede certain activities or tasks. When a gamified learning robot with a 
humanoid appearance presents intelligent features such as voice feedback, communicative dialogs, moving limbs, 
and mood-simulated displays, it is suitable for integrating the STEM curriculum with gamified instruction because 
it could encourage students to achieve learning goals with hand gestures and facial expressions in real-time to 
improve students’ learning motivation and learning attitude in the STEM curriculum and indirectly influence learn- 
ing performance. Gamified educational robots can attract users, elicit learning interests, and make the teaching 
process in the STEM education process more entertaining. The introduction of gamified educational robots can 
help teachers develop curriculum and instruction, and students can practice unfamiliar parts with gamified learn- 
ing robots. In recent years, the widespread integration of educational robots into the classroom to teach specific 
topics has been to appear in the education field in the literature (Chou, 2018). Similarly to these developments, 
educational robots have been integrated into many countries’ curricula and extracurricular activities. However, 
the use of gamified educational robots has not been researched by scholars in the current literature. Despite the 
increased interest in educational robots, there is no current research on using gamified educational robots in the 
classroom to increase student motivation and creativity. From this perspective, gamification elements are integrated 
into the curriculum to make learning more exciting and increase student motivation and participation. Hence, this 
study examined the effects of gamified educational robots on students’ learning motivation and creativity in STEM. 


Literature Review 


The integrated teaching model for science, technology, engineering, and mathematics is called STEM educa- 
tion in the United States. STEM education is stressed because learners can apply the necessary knowledge to solve 
problems in daily life. Many researchers proposed practical courses for STEM learners to solve problems in daily 
life (Estes et al., 20141; Lent et al., 2021). For example, STEM-centered courses using learning robots can increase 
students’ interest in learning and their cognitive and social skills and help them learn innovative technologies 
(Sapounidis & Alimisis, 2020). In parallel with this finding, STEM-centered learning that integrates gamified learning 
robots can have five principles: (1) play and people as the focus of the learning activity, (2) divergent solutions with 
different learning and outcome pathways, (3) students as creators and publishers, (4) sharing and collaboration, 
and (5) engaged learning (Mylonas et al., 2021). Previous research has shown that STEM education can successfully 
integrate robots as learning and teaching tools (Benitti & Spolaér, 2017; Ching et al., 2019), but the use of gamified 
learning robots in STEM practice curricula is very rare. 

A few similar studies used gamified educational robots in STEM education. For example, Dorotea et al. (2021) 
examined computer science teachers’ knowledge, interest, and confidence in using learning robots. They found that 
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teachers’ knowledge, interest, and self-efficacy in using learning robots for teaching are very positive. The results 
also highlight the interest of teachers in using robotics and supporting their students in such learning activities. 
In another research, Rahman (2021) surveyed to examine the experiences of middle school mathematics and sci- 
ence teachers’ experiences with developing and implementing robotics-based instruction. Results indicated that 
the expected learning outcomes for robotics instruction were related to the learning gains observed in traditional 
instruction (content knowledge) and improved student behavior and social, scientific, cognitive, and intellectual 
skills. Garcia-Carrillo et al. (2021) examined teachers’ perspectives regarding an integrated STEM approach used 
in this educational process along with educational coding and robotics in primary education. The results showed 
that teachers viewed the integrated approach of STEM and pedagogical coding and robotics positively despite 
their practical difficulties practically. They emphasized that the STEM approach and its methods benefit students 
and teachers by improving the teaching and learning process. 

Theodoropoulou et al. (2021) systematically reviewed published Greek studies on educational robotics. Their 
analysis revealed that the research promoted the participants’ technological literacy, interest, and excitement. In 
addition, their results show that ER appears to support the development of critical thinking and problem-solving, 
communication, collaboration, creativity, and innovation. In a recent study, Budiyanto et al. (2022) studied the ef- 
fects of a hands-on robotics course for pre-service teachers in a STEM workshop. They found correlations between 
the principles of computational thinking and STEM education and a positive contribution to the learning experi- 
ences of preservice teachers. 

Meschede et al. (2022) examined the effects of an after-school robotics program on students’ interest in STEM 
and the likelihood that participants would pursue STEM in their academic and professional careers. They collected 
data in a 7-year follow-up study of intervention and comparison groups participants. Their results showed that 
college students were significantly more likely to report STEM college-level interests, attitudes, and behaviors than 
comparison students who had not participated in an educational robotics program. In a different study, Boya- 
Lara (2022) implemented an educational robotics program to improve the knowledge and skills of engineering 
students in STEM education. Their results showed that this new teaching tool could promote STEM curriculum 
among engineering students and motivate them to implement it as new educational robotics. Recently, Fegely 
and Tang (2022) examined the effects of educational robotics on prospective teachers’ programming comprehen- 
sion and motivation. Their results showed statistically significant values for prospective teachers’ comprehension 
of programming concepts and motivation toward programming. A recent review by Mou and Li (2022) examined 
the impact of artificial intelligence robots on education. They found 18 articles that met their criteria between 
1999 and 2019. They categorized the studies they examined by topic and found that one of the topics related to 
the studies on robots and education is related to the impact of educational robots with artificial intelligence on 
students’ creativity and motivation. 

Although researchers have conducted some studies on educational robotics and STEM education at the col- 
lege level (for example, Budiyanto et al., 2022), no research examined the effects of gamified educational robotics 
in STEM education on participants’ motivation and creativity. Moreover, scholars have argued the use of gamifica- 
tion and educational robotics in STEM education and reported positive outcomes. For example, Werbach (2014) 
suggested referring to gamification instruction as the process of making the activity more like a game. Recently, 
Kalogiannakis et al. (2021) suggested that gamification design should represent essential elements of a game, 
including game mechanics, dynamics, and framework. In addition, researchers have pointed out that creating a 
new environment, changing the teaching model, and encouraging student motivation are among the factors that 
promote and increase creativity. STEM education enhances students’ creativity (Aguilera & Ortiz-Revilla, 2021). In 
examining the studies on STEM education and robots, students with good learning motivation are more persistent 
and engaged in learning activities in the STEM curriculum, which leads to good learning outcomes. For this reason, 
when designing the STEM curriculum, students’ interest in the subject and its relevance arouse inner drive and 
promote creativity more successfully. It has been pointed out that the STEM curriculum should pay more attention 
to enhancing motivation and providing appropriate incentives in time to achieve positive effects on creativity. 
Because of this reason, the present study hypothesizes that gamified educational robots strengthen interdisciplin- 
ary integration and improve student learning outcomes. Based on the above considerations, this study aimed to 

1. Promote students’ learning motivation and creativity by introducing gamified educational robots into 
the STEM curriculum. 
2. Promote students’ creativity through learning motivation in the STEM curriculum. 
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Research Design 


This study used a quasi-experimental research design to determine the effects of gamified educational 
robots on participants’ learning motivation and creativity in STEM. A pre-test and post-test design were used in 
experimental and control groups for this aim. Before the experiment, both groups were tested using data collec- 
tion instruments. The same instruments were used to determine the effects of instruction after the experiment. 


Participants 


Students from a university in Taiwan were included as participants in the present study. The participants had 
enrolled in an information technology course. They were from the Department of Computer Science & Informa- 
tion Engineering of the National Kaohsiung University of Science and Technology, Taiwan. The participants were 
108 students who registered for the course in the experimental and control groups. The participants attended 
an information technology course for eighteen weeks in the spring semester between February and July of 2022 
teaching year. The experimental and control groups were randomly selected. Each group included 54 students. 
Participants in the experimental and control groups attended the 18-week experimental class and project practice. 
In this study, students with knowledge of tool application and programming were selected to provide a more 
comprehensive basis. The author obtained the necessary permissions from the university where the research was 
conducted. In addition, the authors promised the participants and lecturers that the data obtained from this study 
would not be shared with third parties. 


Procedures 


One experimental and control group included the effects of gamified educational robots on students’ learning 
motivation and creativity. The steps that the researchers followed in the study are as follows. First, 108 students 
were determined as participants in two classes at a college in Taiwan. One of the classes as the experiment and the 
other class as the control group was chosen. Both groups consisted of 54 students. Second, a pre-test on course 
explanation, learning motivation, and creativity was administered during the first week of the semester in which 
the students were taught. Third, the teaching experiment was conducted over sixteen weeks, including fifty min- 
utes each week. Fourth, a post-test on learning motivation and creativity was administered during the last week 
of the teaching semester. Fifth, practical ability was assessed based on the students’ work. Sixth, the researchers 
analyzed the data obtained from the survey instruments. 

According to the above literature, the following variables are examined in this study. 


Figure 1 
Structure Diagram 


Learning Motivation 
1. Attention 
2. Relevance 
3. Self-confidence 
4. Satisfaction 


Experimental Group 
Introduction of gamification 
in educational robots to 
STEM curriculum 


Control Group 
Introduction of conventional 
teaching to STEM curriculum 
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Two major practices of (1) intelligent streetlight and device placement area and (2) smart trash cans were 
implemented in the present study. The STEM teaching model integrating science, technology, engineering, and 
mathematics was used during the activities. STEM education was emphasized because the control group stresses 
knowledge discourse, causing learners to lack the ability to solve problems in daily life. Many researchers propose 
hands-on courses in STEM education for learners to solve problems in daily life (Lei et al., 2020). Various activities 
were designed to provide more development opportunities for students to select their traces and appearance con- 
figuration. The work was done after completely connecting such devices and wires. A gamified educational robot 
was introduced according to the experimental group's course objective, learning content, and game mechanism. 

For the curriculum design, a gamified education robot was used in the experimental group that (1) can be 
controlled by programming or communication technology or precede an appropriate learning activity, (2) whose 
display has ahumanoid appearance, a movable head, hands, and legs, and can simulate a mood, and (3) has voice 
recognition and communicative dialog functions.“Smart City” is the core of the curriculum design, which uses STEM 
interdisciplinary knowledge to make students understand the composition of devices and device components in 
the smart city and the operating principles of these devices and combine them with general knowledge, which is 
no longer the learning of theories but can be combined with daily life. lt enables students to integrate the “smart 
trash cans” and “intelligent streetlights” planned in their project into the knowledge-learning process through 
programming and practicing the learned knowledge and ideas. The course content was based on the modular 
programming and problem-solving practice required for the information technology course. The curriculum design, 
discussed and drafted with teachers from the relevant areas on and off campus, is consistent with information 
technology learning content and includes a discussion of social issues related to information technology (smart 
city) and application and practice of modular programming (smart Arduino intelligent streetlight, smart trash 
can). The programming language used in the course is taught through a block-oriented language (motoBlockly). 

The teacher then guided the students to understand the importance of streetlights and trash cans to develop 
more functions. Researchers used films and videos to introduce students to intelligent streetlights and smart trash 
cans designs in various countries. The control group was instructed by the instructors’ teacher's narration. The 
students were taught the principle of connection, parts, and programming. For the experimental group students 
in the experimental group have game challenges to review course content. On the other hand, the students in 
the control group discussed with each other and followed the teacher's instructions. This study’s implementation 
of STEM activities was conducted over 18 weeks in three stages. A gamified educational robot was used in each 
lesson to foster motivation and assist students with the problems. Researchers noted students’ challenges during 
the activities and sought to facilitate the teaching process. The detailed content and explanation for each stage 
are listed in Table 1. 


Table 1 
Details of Activities in Procedures 


Weeks Activities per each week Brief description of the activities 
1-2 This stage stresses on the intro- The two-week course in this stage is divided into 2 parts. (1) The introduction of a smart city, which 
duction and knowledge of smart was carried forward to find out intelligent stuff in city life. (2) The understanding of basic use and the 
city-related concepts. relevant application as well as familiarizing the user interface. 


The experimental group has a gamified educational robot to induce motivation in order to know the 
objective of the lesson and the use of the gamified educational robot. The control group applies student 
discussion and teacher instruction. 


3-7 This stage focuses on learning (1) To learn electronic devices used for smart cities and understand the principles as well as learn to 
basic applications. program. (2) To learn the operation logic of block-oriented language and the meanings and use of blocks. 
The experimental group has a gamified educational robot for interactive discussion and problem-solving 

in practice. The control group applies student discussion and teacher instruction. 


8-12 This stage stresses on students’ The teacher would simply introduce the concept of the practice work and the use of hole sites on 
application and practice after the density board. The students would self-construct, decorate, and complete in the remaining time. 
the learning in the previous 
two stages. 
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Weeks Activities per each week Brief description of the activities 
13-17 It is the point in this stage to Agamified educational robot would introduce the course objective, learning content, and game mecha- 


have students precede intel- nism to the experimental group. The teacher then guides the students to understand the importance 
ligent streetlight design andthe of streetlights and trash cans to develop more functions. Intelligent streetlights and smart trash can 
application and practice ofsmart designs in various countries in the world are introduced through films and videos. The control group is 
trash cans. simply instructed by the teacher’s narration. 
The students are taught to know the principle of part assembly and connection as well as programming. 
For the experimental group, students in the experimental group have game challenges to review 
course content. The control group, on the other hand, would discuss with each other and accept the 
teacher's instruction. 


18 Learning outcome check The works of students in different groups are evaluated, and the learning motivation and creativity 
post-test is preceded. 


Data Collection Instruments 
Learning Motivation Scale 


The scale we used in this study to measure students’ motivation to learn was the Learning Motivation Scale, 
developed by Lee et al. (2021). They developed this scale according to the ARCS motivation model. The scale con- 
tained four dimensions, including attention (twelve items), relevance (nine items), self-confidence (nine items), 
and satisfaction (six items). It consisted of 36 items in total. Each item was rated on a 5-point Likert scale ranging 
from 1 (strongly disagree) to 5 (strongly agree). The higher scores gained from the scale show a higher motivation 
to learn. Cronbach's alpha was 86 in the scale development study. 


Creativity Scale 


The Creativity Assessment Packet (CAP), developed by Williams (1972), was used to assess student creativ- 
ity. The CAP aimed to evaluate creativity, merely the “creativity tendency scale” for the participants’ works. Many 
researchers have still used the CAP, even in the newest research in recent years. It has been used as a tool for 
evaluating creativity. It has 50 questions. It was scored according to three points for complete compliance, two 
for partial compliance, and one for complete discord. The total score that could be gained was 150 points. There 
are ten reverse questions among them. If a student gets a higher score from the CAP, it demonstrates a higher 
level of creativity. The CAP requires students to complete a series of 12 incomplete figures. The CAP consisted of 
sub-dimensions such as fluency, openness, flexibility, originality, elaboration, and title. Higher scores indicate a 
stronger ability for a particular aspect. Each dimension of the CAP was scored differently according to guidelines 
suggested by the developers. The CAP included instructions for scoring. Researchers used these instructions to 
assess student creativity. 


Data Analysis 


Analysis of variance was applied to determine the difference between gamified educational robots in the 
STEM curriculum in terms of learning motivation and creativity. Regression analysis was used to understand the 
relationship between learning motivation and creativity. An independent samples t-test was used to analyze the 
differences between the two groups. The results showed that the pre-tests for both groups’ prior knowledge, 
learning motivation, and creativity were insignificant (p = .371; p = .533; p = 482). 
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Research Results 
Effects of Gamified Educational Robots in STEM Curriculum on Learning Motivation 


An independent t-test was conducted to determine the effects of the analysis of gamified educational robots 
in STEM education on learning motivation. For this aim, we run the test for each dimension of the motivation scale. 
According to independent t-test analysis, we found significant differences between mean scores between control 
and experimental groups. Table 2 shows higher attention (4.47 > 4.03), relevance (4.52 > 3.81), self-confidence 
(4.44 > 3.92), and satisfaction (4.43 > 3.89) in favor of the use of gamified educational robots in STEM curriculum 
than in the control group. 


Table 2 
Difference Analysis of Gamified Educational Robots in STEM Curriculum on Learning Motivation 


Variable M t p 

Experimental group 4.47 

Attention 2.528 013 
Control group 4.03 
Experimental group 4.52 

Relevance 4.527 .001 
Control group 3.81 
Experimental group 4.44 

Self-Confidence 3.197 .002 
Control group 3.92 
Experimental group 4.43 

Satisfaction 3.104 .003 
Control group 3.89 


Effects of Gamified Educational Robots in STEM Curriculum on Creativity 


We conducted an independent t-test analysis to determine the effects of the analysis of gamified educational 
robots in STEM education on creativity. According to independent t-test analysis, we found a statistical difference 
between the control and experimental groups. This difference is in favor of the experimental group. Table 3 reveals 
higher creativity in the gamified educational robots in the STEM curriculum than in the control group (p < .001). 


Table 3 
Difference Analysis of Gamified Educational Robots in STEM Curriculum on Creativity 


Variable M t p 
ene Experimental group 4.61 ve a 
reativi : : 
j Control group 4.06 


Note.p<01 


Correlation Analysis of Learning Motivation and Creativity 
Table 4 reveals notably positive effects among subdimensions of the motivation scale, such as attention, 


relevance, self-confidence, and satisfaction (8 = 0.311***; 0.265***; 0.297***; 0.160***) and creativity. These results 
mean that the subdimension of motivation has a statistically positive effect on creativity. 
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Analysis of Learning Motivation for Creativity 


Dependent variable— 
Creativity 
Independent variable | 
Learning motivation p p 
Attention 31 001 
Relevance .265*** .001 
Self-confidence 297" .001 
satisfaction .160*** .001 
F 358.541 
Significance 001 
R2 .933 
adjusted R2 .930 


Note. *** stands for p < .001. 


Discussion 


The research results showed that gamified educational robots could effectively enhance learning motiva- 
tion because the educational robot could enhance students’ attraction to the appearance. The educational 
robot provided students with lesson points and reminded students to ask questions. This result shows that a 
gamified educational robot presents higher affinity and interaction with students, as it is not a real person, as 
well as gesture, speech, and expression. When students inquire, the gamified educational robot enhances their 
confidence by allowing them to interact with real people and learn in an interesting process. In sum, the gami- 
fied educational robot allows students to acquire confidence in the learning process and enhance participation 
and motivation in class to acquire satisfaction further. This result is partly similar to the results of Madariaga 
et al. (2023). For example, Madariaga et al. (2023) found that tangible robotics games successfully motivated 
elementary school students to develop collaborative computational thinking with coding problems as part of 
the mathematics curriculum. Furthermore, the present study found a statistically significant difference between 
the control and experimental groups in favor of the experimental class that used gamified educational robotics 
in the STEM classroom. There is a similarity between the study of Madariaga et al. (2023) and this study regard- 
ing increasing the participants’ motivation. In addition, this result is consistent with previous research that has 
shown that educational robotics helps increase preservice science teachers’ motivation as they learn to program 
the robots (Barak & Assal, 2018; Fegely & Tang, 2022; Kim et al., 2015; Kucuk & Sisman, 2018). For example, Fegely 
and Tang (2022) pointed out that learning robotics can improve pre-service teachers’ motivation. Moreover, the 
results related to motivation support the findings of Anwar et al. (2019), who examined a systematic review of 
educational robotics. Their analysis found that, in general, researchers have used educational robotics to enhance 
and promote student creativity, and research was guided by the idea that educational robotics increased stu- 
dent interest and motivation. In their analysis, they found 53 studies on motivation and creativity. Similarly, the 
results are similar to those of Cuellar et al. (2014), who report that unique and innovative educational robotics 
motivated students to be interested in science and technology. 

The experimental group had significantly higher creativity than the control group. This result shows that 
including gamified educational robots positively affected learners’ creativity tendencies. In the briefing cur- 
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riculum design of the gamified educational robot, it is difficult for students to receive solutions from it directly. 
The gamified educational robot often applies speech functions to ask students or have students precede think- 
ing. During the challenge, students were also encouraged when making mistakes, allowing them to receive 
inspiration without destroying their emotions and to attempt unknown affairs. In this case, learners, under the 
assistance of the educational robot, are more enthusiastic, curious, and willing to step into unknown areas for 
exploration and learning. The findings on creativity in this study are very similar to those of Theodoropoulou 
et al. (2021), who revealed that educational robotics supported the development of creativity and innovation 
skills of students at many levels in the majority of the studies they examined. The results related to creativity 
support the findings of Anwar et al. (2019), who conducted a review on educational robotics. Their analyses 
revealed that researchers used educational robotics to enhance and promote students’ creativity. Their results 
show that educational robotics has increased students’ creativity and that many studies have helped to increase 
and promote students’ creativity. 

This study added new insights to the literature by incorporating gamified educational robots into STEM 
lessons to enhance student motivation and creativity. This research shows that gamified educational robotics 
can complement STEM lessons to increase student engagement and learning motivation. Hence, this research 
contributes to the literature by providing robust evidence for using gamified educational robots in the STEM 
classroom. Moreover, this evidence provides a new viewpoint for using gamified learning robots in the STEM 
classroom. 

This study has some limitations. First, the study was conducted with a limited number of students in Taiwan, 
and the experimental teaching lasted eighteen weeks. Since the context in which this study was conducted has 
some unique features, there may be differences in context and contextual factors in further studies. For this rea- 
son, future research should be conducted in other contexts. Second, the activities in this study were developed 
based on the researcher's background and experience with students. Different teaching methods and techniques 
may be considered to combine and use gamified educational robotics in STEM teaching in different studies. 


Conclusion and Implications 


This study examined the effects of gamified educational robots on participants’ learning motivation and 
creativity in STEM. The results showed that gamified educational robots could enhance learning motivation and 
positively influence learners’ creativity. The results also showed that learning motivation has significant effects 
on creativity, and students with high motivation perform better in terms of creativity. The trends in robot tech- 
nology strengthen interdisciplinary STEM teaching and effectively promote students’ learning outcomes. The 
results show that this research provides important practical implications for using gamified educational robotics 
in STEM education to engage participants in learning and support their creativity in STEM-related studies. In 
response to the gradual emphasis on STEM curriculum in education, combining it with practical exercises can 
help students realize the content required on STEM with the knowledge they have learned. Besides increasing 
learning fun, they could apply and create what they have learned to build a personal knowledge structure. 

For this reason, the learning content in technology was chosen as the basis for the course content, and 
the smart city was used as the theme for the STEM practice curriculum, which includes science, technology, 
engineering, mathematics, and the corresponding fields so that students can incorporate electronic elements 
into the life-related course content. This study’s experimental and control groups applied the STEM curriculum 
to lesson design, including participation, exploration, explanation, implementation, deepening, and evaluation. 
Organized and hierarchical methods were used to guide students in constructing knowledge. By integrating 
gamification mechanisms into the STEM curriculum, the learning robot can provide gestures, facial expressions, 
and speech to make the behavior closer to humans so that students would be more willing to communicate 
and ask relevant questions. Gamification mechanisms can make the learning content more engaging. The robot 
provides positive encouragement when students make mistakes in challenges or encounter problems in practice 
to build confidence and motivation. Implementing a STEM activity requires new strategies and expertise for 
instructors. Such implementation depends on the creation of new learning environments that allow participants 
to learn and experience new technologies in STEM education that is new to them. 


https://doi.org/10.33225/jbse/23.22.427 QT 


A35 


Journal of Baltic Science Education, Vol. 22, No. 3, 2023 


GAMIFIED EDUCATIONAL ROBOTS LEAD AN INGREASE IN MOTIVATION AND GREATIVITY IN ISSN 1648-3898 /Print/ 
STEM EDUCATION 

(PP. 42'7-438) ISSN 2538-71 38 /Online/ 
Recommendations 


Regarding the planning of the teaching content, a smart city was chosen as the theme for the design. Con- 
tent for the participants can be closer to students’ lives and apply to them what they have learned in class. In this 
way, students can be better motivated and more easily achieve the teaching objectives set by the teacher. The 
interactive curriculum content with the educational robot in the STEM curriculum is set. The content is designed 
so students can trigger it by clicking on it. They need to open the corresponding web pages through a computer 
to make changes. Thus, the learning robot used in this study includes corresponding speech kits and speech as- 
sistants for development. The researcher proposes to develop this part in the future to make it more intuitive for 
students to use and better present intelligence. 

In this study, gamified learning robots are used in STEM curricula, and the relevant practice is learning content. 
Although the content has been designed and the difficulty level has been considered, students still need help un- 
derstanding the principles of the elements and the logical programming content. In addition, when the teaching 
time is short, many students are interrupted by the learning process when they are about to engage in the prac- 
tice. In this case, the implementation complexity should be simplified in the future. Continuous instruction can be 
necessary, such as two consecutive sessions or club time students can learn wholly and continuously. In addition, 
lecturers who teach STEM subjects should ensure the implementation of formal STEM education and encourage 
students and their colleagues to participate in formal STEM activities (Wang et al., 2021). Second, scholars should 
seek to introduce gamified educational robots, implement them in teaching STEM subjects, and explore how they 
can be integrated into the teaching process to increase interest and motivation in STEM subjects. 


Declaration of Interest 
The authors declare no competing interest. 


References 


Aguilera, D., & Ortiz-Revilla, J. (2021). STEM vs. STEAM education and student creativity: A systematic literature review. Education 
Sciences, 11(7), Article 331. https://doi.org/10.3390/educsci1 1070331 

Bai, S., Hew, K. F., & Huang, B. (2020). Does gamification improve student learning outcome? Evidence from a meta- 
analysis and synthesis of qualitative data in educational contexts. Educational Research Review, 30, Article 100322. 
https://doi.org/10.1016/j.edurev.2020.100322 

Barak, M., & Assal, M. (2018). Robotics and STEM learning: Students’ achievements in assignments according to the p3 task 
taxonomy—Practice, problem solving, and projects. International Journal of Technology and Design Education, 28(1), 121-144. 
https://doi.org/10.1007/s10798-016-9385-9 

Benitti, F. B. V., & Spoladr, N. (2017). How have robots supported STEM teaching? In: Khine, M. (Ed.), Robotics in STEM education 
(pp. 103-129). Springer. https://doi.org/10.1007/978-3-319-57786-9_5 

Boya-Lara, C., Saavedra, D., Fehrenbach, A., & Marquez-Araque, A. (2022). Development of a course based on BEAM robots to 
enhance STEM learning in electrical, electronic, and mechanical domains. International Journal of Educational Technology 
in Higher Education, 19(1), 1-23. https://doi.org/10.1186/s41239-021-00311-9 

Budiyanto, C. W., Fenyvesi, K., Lathifah, A., & Yuana, R. A. (2022). Computational thinking development: Benefiting from educational 
robotics in STEM teaching. European Journal of Educational Research, 11(4), 1997-2012. https://doi.org/10.12973/eu-jer.1 1.4.1997 

Chevalier, M., Giang, C., Piatti, A., & Mondada, F. (2020). Fostering computational thinking through educational robotics: 
A model for creative computational problem solving. International Journal of STEM Education, 7(1), Article 39. 
https://doi.org/10.1186/s40594-020-00238-z 

Ching, YH., Yang, D., Wang, S., Baek, Y., Swanson, S., & Chittoori, B. (2019). Elementary school student development 
of STEM attitudes and perceived learning in a STEM integrated robotics curriculum. TechTrends, 63, 590-601. 
https://doi.org/10.1007/s1 1528-01 9-00388-0 

Chou, P.-N. (2018). Skill development and knowledge acquisition cultivated by maker education: Evidence from Arduino- 
based educational robotics. Eurasia Journal of Mathematics, Science and Technology Education, 14(10), Article 
em1600. https://doi.org/10.29333/ejmste/93483 

Cuellar, F., Penaloza, C., Garret, P., Olivo, D., Mejia, M., Valdez, N., & Mija, A. (2014). Robotics education initiative for 
analyzing learning and child-parent interaction. 2014 IEEE Frontiers in Education Conference (FIE) Proceedings, 1-6. 
https://doi.org/10.1109/FIE.2014.704445 


436 


Wass https://doi.org/10.33225/jbse/23.22.427 


Journal of Baltic Science Education, Vol. 22, No. 3, 2023 


ISSN 1648-3898 /Print/ GAMIFIED EDUCATIONAL ROBOTS LEAD AN INGREASE IN MOTIVATION AND GREATIVITY IN 
STEM EDUCATION 
ISSN 2538-71 38 /Online/ (PP. 427-438) 


Daher, W. (2022). Students’ motivation to learn mathematics in the robotics environment. Computers in the Schools, 39(3), 230- 
251. https://doi.org/10.1080/07380569.2022.2071227 

Dorotea, N., Piedade, J., & Pedro, A. (2021). Mapping K-12 computer science teacher’s interest, self-confidence, and knowledge 
about the use of educational robotics to teach. Education Sciences, 11(8), Article 443. https://doi.org/10.3390/educscil 1080443 

Estes, M. D., Liu, J., Zha, S., & Reedy, K. (2014). Designing for problem-based learning in a collaborative STEM lab: A case study. 
TechTrends, 58, 90-98. https://doi.org/10.1007/s1 1528-014-0808-8 

Fegely, A., & Tang, H. (2022). Learning programming through robots: The effects of educational robotics on pre-service teachers’ 
programming comprehension and motivation. Educational Technology Research and Development, 70, 2211-2234. 
https://doi.org/10.1007/s11423-022-10174-0 

Garcia-Carrillo, C., Greca, |. M., Fernandez-Hawrylak, M. (2021). Teacher perspectives on teaching the STEM approach to educational 
coding and robotics in primary education. Education Sciences, 11(2), Article 64. https://doi.org/10.3390/educsci1 1020064 

Gubenko, A., Kirsch, C., Smilek, J. N., Lubart, T., & Houssemand, C. (2021). Educational robotics and robot creativity: An 
interdisciplinary dialogue. Frontiers in Robotics and Al, 8, Article 662030. https://doi.org/10.3389/frobt.202 1.662030 

Hanif, S., Wijaya, A. F. C., & Winarno, N. (2019). Enhancing students’ creativity through STEM project-based learning. Journal of 
Science Learning, 2(2), 50-57. https://doi.org/10.17509/jsl.v2i2.13271 

Kalogiannakis, M., Papadakis, S., & Zourmpakis, A. |. (2021). Gamification in science education. A systematic review of the literature. 
Education Sciences, 11(1), Article 22. https://doi.org/10.3390/educsci11010022 

Kim, C., Kim, D., Yuan, J., Hill, R. B., Doshi, P., & Thai, C. N. (2015). Robotics to promote elementary education pre-service teachers’ 
STEM engagement, learning, and teaching. Computers & Education, 91, 14-31. https://doi.org/10.1016/j.compedu.2015.08.005 

Kucuk, S., & Sisman, B. (2018). Pre-service teachers’ experiences in learning robotics design and programming. Informatics in 
Education, 17(2), 301-320. https://doi.org/10.15388/infedu.2018.16 

Lee, S., Shih, M., Liang, J., & Tseng, Y. (2021). Investigating learners’engagement and science learning outcomes in different designs 
of participatory simulated games. British Journal of Educational Technology, 52(3), 1197-1214. https://doi.org/10.1111/ 
bjet.13067 

Lei, H., Chiu, M. M., Li, F., Wang, X., & Geng, Y.-j. (2020). Computational thinking and academic achievement: A meta-analysis 
among students. Children and Youth Services Review, 118, Article 105439. https://doi.org/10.1016/j.childyouth.2020.105439 

Lent, D. D., Estes, K. M., & Hansen, A. K. (2021). Increasing faculty involvement in the undergraduate interdisciplinary learning 
experience. Integrative and Comparative Biology, 61(3), 1002-1012, https://doi.org/10.1093/icb/icab109 

Leonard, J., Buss, A., Gamboa, R., Mitchell, M., Fashola, O. S., Hubert, T., & Almughyirah, S. (2016). Using robotics and game design 
to enhance children’s self-efficacy, STEM attitudes, and computational thinking skills. Journal of Science Education and 
Technology, 25, 860-876. https://doi.org/10.1007/s10956-016-9628-2 

Madariaga, L., Allendes, C., Nussbaum, M., Barrios, G., & Acevedo, N. (2023). Offline and online user experience of gamified robotics 
for introducing computational thinking: Comparing engagement, game mechanics and coding motivation. Computers & 
Education, 193, Article 104664. https://doi.org/10.1016/j.compedu.2022.104664 

Meschede, T., Haque, Z., Warfield, M. E., Melchior, A., Burack, C., & Hoover, M. (2022). Transforming STEM outcomes: Results from 
a seven-year follow-up study of an after-school robotics program's impacts on freshman students. School Science and 
Mathematics, 122(7), 343-357. https://doi.org/10.1111/ssm.12552 

Mou, X., & Li, R. ¥. M. (2022). The impact of artificial intelligence educational robots in the field of education: A PRISMA review. In 
R.Y. M. Li, K.W. Chau, D. C. W. Ho (Eds.), Current State of Art in Artificial Intelligence and Ubiquitous Cities (pp. 63-77). Springer. 
https://doi.org/10.1007/978-981-19-0737-1_4 

Mylonas, G., Paganelli, F., Cuffaro, G., Nesi, I., & Karantzis, D. (2021). Using gamification and loT-based educational tools towards 
energy savings-some experiences from two schools in Italy and Greece. Journal of Ambient Intelligence and Humanized 
Computing, 1, 1-20. https://doi.org/10.1007/s12652-020-02838-7 

Rahman, S.M.M. (2021). Assessing and benchmarking learning outcomes of robotics-enabled STEM education. Education Sciences, 
11(2), Article 84. https://doi.org/10.3390/educsci1 1020084 

Reyes, G. E. B., Lopez, E., Ponce, P., & Mazon, N. (2020). Role assignment analysis of an assistive robotic platform in a high school 
mathematics class, through a gamification and usability evaluation. International Journal of Social Robotics, 13, 1063-1078. 
https://doi.org/10.1007/s12369-020-00698-x 

Sailer, M., Hense, J., Mandl, J., & Klevers, M. (2013). Psychological perspectives on motivation through gamification. /nteraction 
Design and Architecture(s) Journal, (19), 28-37. 

Sailer, M., & Sailer, M. (2021). Gamification of in-class activities in flipped classroom lectures. British Journal of Educational Technology, 
52(1), 75-90. https://doi.org/10.1111/bjet.12948 

Sapounidis, T., & Alimisis, D. (2020). Educational robotics for STEM: A review of technologies and some educational considerations. 
In E. Oldham, A. S., Afonso, F., Viseu, L., Dourado, M. H. Martinho (Eds), Science and mathematics education for 21st century 
citizens: Challenges and ways forward (pp. 167-190). Nova Science Publishers. 

Sanchez, D.R., Langer, M., & Kaur, R. (2020). Gamification in the classroom: Examining the impact of gamified quizzes on student 
learning. Computers & Education, 144, Article 103666. https://doi.org/10.1016/j.compedu.2019.103666 

Theodoropoulou, I., Lavidas, K., & Komis, V. (2021). Results and prospects from the utilization of educational robotics in Greek 
Schools. Technology, Knowledge and Learning, 1-16. https://doi.org/10.1007/s10758-021-09555-w 


https://doi.org/10.33225/jbse/23.22.427 LQ 


437 


438 


Journal of Baltic Science Education, Vol. 22, No. 3, 2023 


GAMIFIED EDUCATIONAL ROBOTS LEAD AN INGREASE IN MOTIVATION AND GREATIVITY IN ISSN 1648-3898 | jprint/ 
STEM EDUCATION 
(PP. 427-438) ISSN 2538-7138 /Online/ 


Wang,N., Tan, A.L., Xiao, W. R., Zeng, F., Xiang, J., & Duan, W. (2021). The effect of learning experiences on interest in STEM careers: 
A structural equation model. Journal of Baltic Science Education, 20(4), 651-663. https://doi.org/10.33225/jbse/21.20.651 
Werbach, K. (2014). (Re)defining gamification: A process approach. In A. Spagnolli, L. Chittaro, L. Gamberini (Eds.), Persuasive technology 
- Persuasive, motivating, empowering videogames (pp 266-272). Springer. https://doi.org/10.1007/978-3-319-07127-5_23 
Williams, F. E. (1972). Encouraging creative potential. Educational Technology Publications. 


Received: January 04, 2023 Revised: March 28, 2023 Accepted: April 30, 2023 


Cite as: Chen, T.-I., Lin, S.-K., & Chung, H.-C. (2023). Gamified educational robots lead an increase in motivation and creativity 
in STEM education. Journal of Baltic Science Education, 22(3), 427-438. https://doi.org/10.33225/jbse/23.22.427 


Tan-I Chen PhD, Center for Innovative Research on Aging Society, National Chung 
Cheng University, University Rd., Minhsiung, Chiayi 621301, Taiwan. 
E-mail: dychen115@gmail.com 


Shih-Kai Lin PhD Candidate, Department of Business Administration, National 
Yunlin University of Science and Technology, Taiwan. 
E-mail: sklin1 105@yahoo.com.tw 


Hung-Chang Chung PhD Candidate, Department of Business Administration, National 
(Corresponding author) Yunlin University of Science and Technology, Taiwan. 
E-mail: d10522006@yuntech.edu.tw 


Ass https://doi.org/10.33225/jbse/23.22.427 


@® 6) This is an open access article under the 
(c) Creative Commons Attribution 4.0 
BY NC 


International License 


TYPES OF QUALITY AND EQUAL 
SCIENCE EDUCATION SYSTEMS 
AND THE DEVELOPMENT OF 
EQUALITY IN EDUCATION 


Tao Jiang, 
Hong-wei Jin, 
Wen-jie Ma, 
Ji-gen Chen, 
Ling-min Yuan 


Introduction 


Science education needs to be more egalitarian to address anti-intellec- 
tualism, gender discrimination, and globalization effectively. The conviction 
that science is a theoretical system for explaining nature is not unbreakable. 
Anti-science happens all over the world (Thorp, 2019). Every anti-science adult 
means there was a minor who wandered out of science in the past. As a result 
of academic failure, many disadvantaged students do not believe that science 
is relevant to them (Next Generation Science Standards, 2013, p. 29). But the 
responsibility does not lie with them (McGee & Martin, 2011). Traditionally, 
the education system has tended to ignore the potential of disadvantaged 
students (such as immigrants, people of color, and children from low-income 
families) to learn science (Next Generation Science Standards, 2013, p. 26). 
Sadly, disadvantaged students seem to accept this identity (Kang, 2022). Only 
by significantly raising the academic achievement of disadvantaged groups 
in science can the number of pessimists and even opponents of science in 
our future society be reduced. Science education also neglects to be fair to 
women (Russell, 2020; Ten Hagen et al., 2022). If this situation continues, it 
will be difficult for science and the scientific community to avoid its adverse 
effects (Next Generation Science Standards, 2013, pp. 26-28; Russell, 2020). 
Furthermore, the accelerated process of globalization has led to a greater 
demand for highly skilled labor in society. When society has reached its limits 
in tapping the human resources of its advantaged groups, it has to turn to 
train disadvantaged groups to become skilled STEM laborers (Next Genera- 
tion Science Standards, 2013, p. 26; Philip & Azevedo, 2017). 

Ethics demands science education to increase equality. Science educa- 
tion is designed according to the stereotype of serving the academic elite 
(McGee & Martin, 2011; McNutt, 2020). Even the science museum, a place 
for science education outside of school, reflects the cultural practices of the 
middle class (Feinstein, 2017). As a result, science education is one of the 
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causes of the growing inequality gap between people (McNutt, 2020). Some science education systems are “am- 
blyopic” and do not see disadvantaged students. In such a system, education breaks away from its goal of serving all 
students and becomes subordinate to efficiency. Science education that is “indifferent to disadvantaged students” 
sets the stage for academic failure and discrimination. Even if we attribute the inequality in science education to 
the past, we cannot deny that the time has come to address it (Philip & Azevedo, 2017). 

Inequality in science education extinguishes capital-disadvantaged students’ hopes of success. In the United 
States, blacks, indigenous, and people of color are disadvantaged groups (BIPOCs, McNutt, 2020). In China, they are 
farmers and people who have migrated to the cities to work as casual laborers. Children with inferior and superior 
family capital typically live in different communities and attend diverse schools, as if fated to follow different life 
paths (Barabino, 2020). The lack of interest in science is the rebellion of disadvantaged students against an unequal 
education. There is a growing voice in the science community that the time has come for science education to 
change its systematic discrimination against capital-disadvantaged students (Barabino, 2020; McNutt, 2020; Next 
Generation Science Standards, 2013, pp. 25-29; Russell, 2020; Thorp, 2019). In the United States, some science 
teachers have changed their teaching practices. They have advanced educational equality by integrating science 
practice with the culture of people of color (Kang, 2022; Shea & Sandoval, 2020). It reflects an improved climate of 
educational equality on a small scale (in one disadvantaged school or several poor communities). Equality in science 
education means a lot to individuals and society. Only when science education is equal will disadvantaged students 
give up slacking off. It is then that academic success will not be the exclusive honor of capital-advantaged students. 


Literature Review 
Educational equity and social justice 


The notion of equity is the product of the concept of justice. Justice is a pluralistic concept in political phi- 
losophy, and the utilitarian, liberal, and neoliberal views of justice impact people's opinions on educational equity. 
A person with a utilitarian view of justice believes that maximizing educational productivity is educational equity 
(L. C. Li & Wang, 2014). A liberal justice advocate believes that educational equity means the most disadvantaged 
students benefit the most (Rawls, 2005, p. 80, first published in 1971). A neo-liberal justice advocate believes 
that free competition between schools and students’ free choice of schools is educational equity (Bifulco & Ladd, 
2006). People live together, so diverse views of educational equity coexist and are reflected as a social reality. In 
Estonia, for example, the nation’s implementation of free education and the allocation of financial subsidies to the 
poorest students is what liberals see as fair (Lees, 2016). The country also opens private schools, which account for 
10% of the total schools, and allows students to receive paid education (Lees, 2016), which is equity in the eyes of 
neo-liberalism. China’s pursuit of high performance in science education has set a group of high-quality schools 
apart from the competition. Mainland China was the only education system where students scored at level four 
in science on the 2018 Program for International Student Assessment (PISA) test. It outperformed second-placed 
Singapore in science by 39 points (Schleicher, 2019). From a utilitarian perspective, this is educational equity. In 
recent years, China has allocated central financial funds to fill the gap in rural education funds in various provinces 
(State Council, 2012). The purpose of this measure is to compensate capital-disadvantaged students. It reflects 
a liberal view of justice and suggests that Chinese society has changed its monolithic utilitarian perspective on 
the issue of educational equity. In summary, the notion of equity stems from the concept of justice, and multiple 
understandings of justice can coexist and influence societal and individual beliefs about educational equity. 


Meeting quality and equality in education is equity 


Equality is the basis of equity. Equality is a collective and objective concept, while equity is an individual and 
subjective concept (Volckmar, 2019). Equality is a quantifiable, rational judgment that implies equal treatment for 
all (Clarke, 2014; Volckmar, 2019). Educators follow two routes to refine the definition of equality. One is to define 
three types of equality from the perspective of political philosophy. Equality of opportunity refers to the equal 
rights granted by law to everyone to access primary, secondary, and higher education (Espinoza, 2007). “Equality 
on average across social groups’ refers to the balanced distribution of educational resources among different so- 
cial groups (Espinoza, 2007; Volckmar, 2019). “Equality for all” means that each person receives equal educational 
resources (Carlson, 1983). Thus, equal distribution of educational resources for all students (e.g., the same amount 
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of money per student for education) is equality for all (Allbright et al., 2019; L. C. Li & Wang, 2014). The definition 
of equality from a political philosophy perspective has a distinctly equal distribution character. It emphasizes that 
people are treated identically when accessing educational opportunities and resources. The second is to define 
three types of equality based on a person's school experience. These include equality of opportunity, equality of 
process, and equality of outcome. The nation has enacted laws or educational policies to decrease gaps among 
people in terms of educational opportunities, curriculum learning, and learning outcomes. The definition of equality 
of opportunity is similar to the previous one. It is educational opportunity equality for students from low-income 
families to attend the same school as advantaged students, as well as for students of different races and ethnicities 
to enter the same school without social segregation (Dayton & Dupre, 2004). The teachers’ love and guidance for 
students do not vary according to their performance, family background, race, or ethnicity, which is the equality 
of the educational process (OECD, 2018, p. 9; Savage, 2013). Students achieve similar academic results regardless 
of gender, family capital, race, or ethnicity, indicating equality of educational outcomes (Zhao et al., 2012). The 
public can only perceive equity based on equality. In any case, there is no educational equity when the interests of 
the weak are compromised by severe inequality. For example, the lack of educational resources in schools, which 
makes students drop out or perform poorly, is an example of educational inequity caused by educational inequality 
(Castano et al., 2019; Cuesta & Madrigal, 2014; Gewirtz, 1998; M. Li et al., 2022). 

There may not necessarily be educational equity even with educational equality, for example, in contexts 
where all children receive poor-quality teaching. Policies that aim for equality can sometimes result in waste or 
resources that cannot meet the needs (Allbright et al., 2019). Scholars refer to this as the narrowness of equality 
(Rizvi & Lingard, 2010, p. 156). This condition can harm quality. The No Child Left Behind Act reflects a policy drift 
in the United States: a shift from equality to quality (Griffen, 2022). The reason for this shift is that economists 
believe that educational inequality can only be addressed by improving the quality of teaching (Griffen, 2022). 
Quality is a subordinate concept of educational effectiveness. Quality means that teachers teach effectively. It is 
difficult to have quality teaching when science teachers lack scientific knowledge and are afraid to teach science 
(Lamanauskas, 2022). It has an expanded connotation when quality is used as a subordinate concept to educational 
equity. The teaching is also of low quality when teachers only consider the needs of some students, leading to 
resistance from others (Martin, 2019; Webb & Radcliffe, 2016). Student performance in standardized tests is often 
viewed as a primary quality indicator (Gewirtz et al., 2021). Low-quality teaching is not in the interests of anyone 
and therefore is not equitable. Quality is, therefore, also the basis of equity. Equality and quality depend on each 
other to become educational equity. 


Educational practice of balancing equality and quality 


The difficulty of education practice is that equality and quality are often in conflict. Under the PISA shock, 
Australia publishes school results through the Myschool website to motivate weak schools to improve the quality 
of teaching (Loughland & Sriprakash, 2016). Finland and Sweden enhance school effectiveness by strengthening 
competition (Varjo et al., 2018). The Sage School District in California provides some schools with more material 
and human resources than others (Allbright et al., 2019). These measures favor quality but can undermine equality. 
There are winners and losers in the competition. The winning schools can raise the price of estates in neighbor- 
ing communities and segregate disadvantaged students (Schleicher, 2019). That worsens educational equality. 
Currently, between the elitist tradition of education and equality, there is a voice that regards equality as the core 
value of educational equity (Nachbauer & Kyriakides, 2020; Van Damme & Bellens, 2017). However, it is worth being 
wary that practices conducive to equality may harm the quality of education. Japan and Korea have implemented 
teacher mobility systems that favor equality in education (Fu et al., 2015; OECD, 2018, p. 41). However, this system 
is almost exclusive to Japan and Korea. China praises its value of promoting equality but does not draw on this sys- 
tem. It is because the instability of school human resources may be detrimental to its teaching quality. The practice 
of educational equity thus requires balancing equality and quality. It means neither harming quality for the sake 
of equality nor harming equality for the sake of quality. Estonia, for example, has a comprehensive school system 
and provides financial assistance to disadvantaged students to guarantee equality in education (Lees, 2016). At 
the same time, its curricula and examinations are of a high standard to reinforce the quality of education (Tucker, 
2015). Equality and quality are balanced, rather than sacrificing one for the other, thus advancing equity in educa- 
tion. At this point, the academic performance of both disadvantaged and advantaged students improves, and the 
disadvantaged students’ performance improves even more. 
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Research Problem 


Assessing equality in science education is difficult. The regression analysis has long been a typical data analysis 
method of determining equality in education (Nachbauer & Kyriakides, 2020). It uses a coefficient of determina- 
tion (R-squared) to indicate the strength of the association between family capital and achievement. In the 2018 
PISA test, education systems with a coefficient of determination of less than 10% were considered equal, while 
greater than 15% indicated low equality (OECD, 2019, p.60). However, there is a logical flaw in judging educational 
equality by the R-squared. 

There is a logical defect in using a decreasing coefficient of determination as an increase in educational equal- 
ity. As shown in Figure 1, the oblique line is the best-fitting line. The left side of the horizontal axis is the area of 
capital disadvantage, and the right is the advantaged area. When all data points (each representing one student) 
fall on the fitted line, capital explains 100% of the variance of scores according to the principle of ordinary least 
squares (Lewis-Beck, 1980, p. 22). 


Figure 1 
Educational Equality Interpreted by the Coefficient of Determination 


Score 


Capital 


As shown in Figure 1, students moving from the fitting line to zones A or B can increase the unexplained 
deviation attributed to random error and reduce the strength of the capital-score connection (Mendenhall et al., 
2013, p. 498; Vittinghoff et al., 2012, p. 42). If one were to take the perspective of R-squared, one would argue that 
educational equality increases. However, there is a logical fallacy here. Only migration to Zone A increases equality 
in education as the achievement of capital-disadvantaged students rises. On the other hand, migration to Zone B 
decreases educational equality because the academic performance of capital-disadvantaged students decreases. 
As shown in Figure 1, the migration of students from the fitting line to area C can also reduce the strength of the 
capital-score connection. However, as the achievement gap between students with superior and inferior family 
capital increases, it is also an example of reduced educational equality. In conclusion, there are logical flaws in us- 
ing a decreasing R-squared as evidence of increased educational equality. 

There is a logical defect in using an increasing coefficient of determination as educational equality reduction. 
For example, when students move from Zone B back to the fitting line, the economically disadvantaged students 
narrow the achievement gap with the advantaged students. Educational equality is increasing rather than decreas- 
ing. In addition, several scenarios can cause biased estimates of the coefficient of determination. These include that 
residuals do not comply with independence, normality, and homoscedasticity; multicollinearity among predictors; 
and missing key predictors from the regression equation (George & Mallery, 2020, pp. 193-220; Lewis-Beck, 1980). 
Given the shortcomings of the coefficient of determination, it is necessary to propose new data analysis methods 
to analyze educational equality issues. 
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Research Focus 


The study focused on assessing equality in science education. According to the regression analysis, Finland 
does not rank ideal in science education equality (R? = .105, see Table II.B1.2.4, OECD, 2020a). If this is the case, it 
would suggest that the “quality” pole of education has oppressed the “equality” pole, resulting in a regression in 
equality. For Finland, it is time to consider how to reconcile quality and equality. For other countries, it matters to 
find new learning objectives to guide the development of their science education. 

The study focused on the types of equitable science education systems. Quality and equality are educational 
equity. Equity in science education has been achieved in some developed countries or economies. Traditionally, 
science education has been studied using variable-centered data analysis methods such as regression analysis. 
This approach sees the connections between variables but not the “people.” That makes events a black box. People 
know that there is an equitable science education system but do not know how it is formed. To analyze the types 
of equality in equity science education systems with a new person-centered data analysis method is to expose 
the structure of the “black box.’ 


Theoretical Perspective, Research Aim, and Research Questions 


The difference principle is a theoretical perspective for analyzing issues of educational equity. Equality and 
justice are the core concepts in understanding the difference principle. Rawls (2005) used the example of “dividing 
a cake” to demonstrate how egalitarian distribution can be accomplished (p. 85). Even under a system of equal 
distribution of educational resources, advantaged students still gradually widen the performance gap between 
them and disadvantaged students. Rawls (2005) called this phenomenon the negative marginal contribution of 
the advantaged group to the disadvantaged group (p. 79). For the sake of justice, the education system should 
avoid this phenomenon. Rawls has proposed the difference principle to address this issue. This principle requires 
society to take measures to benefit the least advantaged groups the greatest (Rawls, 2005, p. 83). This study be- 
lieved that the difference principle guided the practice of science education systems in raising the achievement 
of disadvantaged students. 

This study aimed to open the black box of an equitable science education system. It was a two-step process. 
The first step was to know the pathway from low equality to equality; the second was to find out how equality is 
improved. Quality and equality are the reform goals of science education in all countries. The significance of open- 
ing the “black box" is to point out the way for science education in various countries to reach its goal. 

The research questions were: 

1. What educational outcomes have students with different family capital achieved in a quality but low- 
equality science education system? 

2. What types of equality were present in equitable science education systems? 

3. What were the differences in the probability of academic success in science between various student 
groups in the above two kinds of science education systems? 

The answers to the above questions give clues to the path from low equality to equality and the continuous 
improvement of equality. 


Research Methodology 
General Background 


The documentary method was used. Data were collected by the PISA 2018 project team. The PISA test con- 
cerns the well-being of students. It surveyed secondary school students’ family capital through questionnaires 
while measuring their reading, mathematics, and science performance with cognitive items. Student achievement 
reflects the quality of a school and can also indicate whether educational policies support the improvement of 
school effectiveness (OECD, 2020b, p. 39). From the fourth test in 2009 to the seventh test in 2018, the PISA project 
team has released several volumes of reports after each test. The analytical contents of volumes one and two are 
fixed. Volume one analyzes the quality of education in each country or economy based on student achievement. 
The second volume assesses the equity of education systems based on the connection between student family 
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capital and achievement. Participants were anonymized, and their family capital and achievement were recorded 
in the student data set (OECD, 2020c). 

Six science education systems with quality but low equality appeared in the 2018 PISA test. They were the 
control group, and their R-squared ranges between .159 and .201 (see Table II.B1.2.4, OECD, 2020a). Analyzing the 
educational outcomes of capital-disadvantaged and advantaged students in such systems opened the systems’ black 
box. Nine systems with equality and quality in science education emerged from the 2018 PISA test. The strength of 
their family capital and science achievement connections (R-squared) ranged from .016 to .10 (see Table II.B1.2.4, 
OECD, 2020a). It provided a rich sample pool of equitable science education systems. The significant variation in 
R-squared predicted that students in these systems received different types of equality. The discovery of various 
forms of equity in science education gave an idea of how equality has increased. These nine education systems 
were the experimental group. Comparing the equality status of the control group with that of the experimental 
group showed the path from low equality to equality. 

This study used the achievement matrix to assess educational equality. That is a person-centered data analysis 
method that focuses on “Whose educational benefits are guaranteed?” The indicator used in the regression analysis 
to evaluate equality is the coefficient of determination, whereas the statistics used in the achievement matrix are 
relative error and conditional probability. 

Disadvantaged, most disadvantaged, and successful students had different meanings in the following text. 
This study defined groups with family capital below the median as “disadvantaged students.’ The opposite was 
true for “advantaged students.” Groups with family capital below the lower quartile were the most disadvantaged 
students, while students whose family capital was above the upper quartile were the most advantaged. Students 
with scores above the upper quartile were called “success students.’ These students are the primary source of the 
nation’s future STEM workforce (Tai et al., 2006). 


The Achievement Matrix 


The achievement matrix used family capital as a row variable and academic achievement as a column variable. 
Students were divided into four levels according to their family capital. Students in each family capital level were 
then divided into four grades according to academic achievement to obtain the number of students in each element. 


Ni1 M42 «N43 «N44 
N21 N22 N23 N24 
N31 132 M33 N34 
Nay N42 N43 N44 


Achievement matrix = 


Where n,, represented the number of students whose family capital and academic performance were both 
below the 25% percentage level, n,, represented the number of students whose family capital was below the 25% 
percentage level and whose academic performance was between the 25% and 50% percentage levels, and so on 
with the other elements. The upper eight elements represented disadvantaged students, whereas the bottom 
eight represented advantaged ones. 

The achievement matrix analysis technique is designed based on the concept of a veil of ignorance (Rawls, 
2005, pp. 136-141). Behind a veil of ignorance, nature does not perceive any difference between high and low 
family capital. Students at a particular family capital level are assigned equally to the four cells in the row of that 
family capital level in the achievement matrix. Setting the total number of students at N gives an expectation of 
.0625N for each element. 


The Relative Error 


This study defined students on the diagonal as regular students, reflecting the degree of consistency between 
family capital and achievement. Elements n,, and n,, also represented disadvantaged students. The other identity 
for individuals in elements n,, and n,, was that of advantaged students. The relative error for regular students was 
the absolute value of the actual number of regular students minus the expected number, divided by the expected 
number. Since inequality existed in every education system, this difference (the numerator part of the relative error 
formula) was always positive. 
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In the case of the most disadvantaged students, those in elements N,,,,and n,, achieved academic achieve- 
ment grades that exceeded their family capital levels. The same was true for students in n,, and n,,. The area in 
which n,,,n,,,and n,, were located was in the low-score area. If education were completely unequal, disadvantaged 
students would be grouped in this partition. The elements n,,, 0,5, 1,4, and n,, represented students who mi- 
grated from the low-score division. The more students in these five elements, the better the educational benefits 
for disadvantaged students. The relative error for disadvantaged students was the absolute value of the actual 
value of these five elements minus their expected value, divided by the expected value. This difference has always 
been a negative number. Due to educational inequality, the actual number of students in elements Nyy M437 Dg Dy 3r 
and n,, was always less than the expected number. 

For advantaged students, elements n,,, n,,, and n,, were located in the high-score area. If education were 
completely unequal, they would all be in this section. Elements n,,,n,,, 0,,,,,,and n,, represented students who 
migrated from the high-score division. The more students there were in these five elements, the smaller the per- 
formance gap between them and the disadvantaged. The relative error for advantaged students was the absolute 
value of the difference between the actual value of these five elements and their expected value, divided by the 


expected value. This difference was always negative due to educational inequality. 
The Conditional Probability 


Students with various family capital had different odds of academic success in science. The probability of 
achieving level 4 in science when family capital was at level 1 (P [score = 4 | capital = 1]) represented the friendli- 
ness of science education to the most disadvantaged students. Similarly, P [score = 4 | capital = 4] demonstrated 
how science education could benefit the most advantaged students. The closer P [score = 4 | capital = 4] and P 
[score = 4 | capital = 1] were, the better the educational equality. 


Person-centered 


The achievement matrix sees people. Each student had a fixed position in the matrix based on their family 
capital and scores. Relative errors and conditional probabilities measure educational outcomes. For example, the 
decreasing relative errors for disadvantaged and advantaged students indicated that the achievement gap had 
narrowed. Conditional probabilities determined the degree of educational equality by analyzing the difference in 
the chances of academic success between disadvantaged and advantaged students. 

Relative error is the primary indicator of educational equality. It was calculated with all the information in 
the achievement matrix. The conditional probability was only related to the elements of the fourth column of the 
matrix. The conditional probability is a secondary indicator of education equality, and its significance is to explain 
the original family capital structure of the country’s future STEM workforce. 


Sample 


The experimental group samples were from quality and equal education systems. Based on the PISA criteria, 
a score above the average of OECD countries was considered good quality, and a coefficient of determination 
below 10% was considered equality in education (OECD, 2019, p. 60). In the 2018 test, Australia, Canada, Estonia, 
Hong Kong (China), Japan, Korea, Latvia, Macao (China), and Norway met the criteria (see Table II.B1.2.4, OECD, 
2020a; Schleicher, 2019). These nine systems are the objective references for science education. PISA has defined 
a coefficient of determination of 15% or more as low equality in education (OECD, 2019, p. 60). The control group 
samples were from six science education systems with good quality but lower equality. They were Belgium, the 
Czech Republic, France, Germany, Portugal, and Switzerland (see Table II.B1.2.4, OECD, 2020a; Schleicher, 2019). 


Measurement and Procedures 
The achievement matrix was created using two variables: family capital and science scores. Science scores were 
divided into eight levels, from below 261 to above 708; students achieving a score of 633.33 or above were called 


top performers in science (OECD, 2019, p. 152; Schleicher, 2019). The PISA test has used family assets to represent 
economic capital, parents’ occupational status to describe social capital, and parents’ educational status to indicate 
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cultural capital to obtain the student's family capital (OECD, 2020d). This variable was standardized and named an 
index of economic, social, and cultural status (abbreviated as ESCS; OECD, 2020d). The student data set recorded 
the ESCS and science scores of the participants (OECD, 2020c). The ESCS had missing values, and the score had no 
missing values. Records with missing values were removed, then the analysis procedure was performed. 

The analysis process was divided into two steps. The first was to calculate the relative error based on the 
achievement matrix. Descriptive statistics were performed using SPSS 25 to obtain three quartile values for ESCS 
and scores. Then the data was saved in spreadsheet format. The SUMPRODUCT() function in the spreadsheet 
returned the number of students in elements n,,~n,,. Three types of relative errors were calculated according to 
the previous definition. The second step was to derive the joint probability based on the achievement matrix and 
then calculate the conditional probability. There were sixteen conditional probabilities in total. This study was con- 
cerned with four conditional probabilities representing academic success in science. The probability that ESCS = 1 
occurred simultaneously with score = 4 was calculated from the value of element n,,. The marginal probability of 
ESCS = 1 was .25.P [score = 4 | ESCS = 1] was obtained by dividing the joint probability by the marginal probability. 
The process for calculating the other three conditional probabilities was the same. P [score = 4 | ESCS = 1] and P 
[score = 4| ESCS = 2] are significant for society. They indicate whether the nation’s intention to train disadvantaged 
groups to become skilled STEM workers is likely to be realized. 


Data Analysis 


As shown in Table 1, the sample has good data quality. The percentage of missing data ranged from .24% to 
14.58%. Samples with more than 5% missing data were Germany (14.58%), Australia (10.06%), Canada (5.13%), 
and Portugal (5.06%). Samples with no more than 15% missing data do not cause significant bias in the analysis 
results (Creswell, 2014, p. 202). Top performers and standard deviations were calculated using the original dataset. 
There were four samples where top performers accounted for more than 10% of the total. They were Japan, Macao 
(China), Estonia, and Korea. These four countries or economies occupied the top four positions in the PISA science 
performance ranking among the fifteen samples (Schleicher, 2019). The quartiles in Table 1 are calculated after 
deleting the missing values. 


Table 1 
Sample Size of Participating Countries or Economies and Descriptive Statistics for ESCS and Science Scores 


Goantiedi Sample size Percentiles ESCS Percentiles_Score Top per- 
economics formers % se 
Valid Missing 25 50 75 25 50 75 

Belgium 8312 163 -.562 188 842 436.8 511.2 572.5 7.0 93.6 
Czech Republic 6911 108 -.146 -.162 626 447.2 516.5 583.1 9.5 92.8 
France 6176 132 -.696 -.025 629 416.8 492.3 557.6 5.1 94.7 
Germany 4656 795 -.198 -.036 119 444.2 519.7 586.3 8.9 98.6 
Portugal 5632 300 -1.27 -.381 671 429.1 495.2 557.2 4.0 86.7 
Switzerland 5705 117 -.597 048 143 430.3 497.3 562.4 6.2 90.6 
Australia 12813 1460 -.302 441 1.03 435.5 509.1 575.1 8.3 95.2 
Canada 21490 1163 -.192 AT9 996 447.7 512.4 575.5 8.0 89.5 
Estonia §202 114 -.502 144 144 473.3 532.7 587.2 11.2 82.6 
Hong Kong (China) 5839 198 -1.24 -.579 212 469.8 527.1 575.6 BS) 78.9 
Japan 6055 54 -.620 -.071 437 468.1 534.7 593.9 11.8 87.5 
Korea 6626 24 -.445 156 651 458.7 §27.3 586.9 10.6 91.9 
Latvia 5187 116 -.666 .056 659 432.3 486.9 540.5 2.0 77.4 
Macao (China) 3766 9 -1.17 -.571 162 494.2 548.6 598.6 11.6 (e3) 
Norway 5612 201 028 666 14 428.1 496.1 558.9 5.3 93.2 


Note. SD = the standard deviation of science scores. 
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Research Results 
Science Education Systems with Low Equality 


Six samples from the control group fell into this category. Table 2 gives information on the three types of 
relative errors for the fifteen samples. The characteristics of the low-equality education system were that the rela- 
tive error for advantaged and disadvantaged students exceeded 19%, and the relative error for regular students 
approached or exceeded 50%. The reason was that the values of elements n,, and n,, were particularly large. The 
scores of the most disadvantaged students concentrated in the region of the lower quartile, and the scores of 
the most advantaged students fell into the upper quartile area. In the French sample, for example, n,,= 752 and 
n,, = 768. The value of element n,, was 1.95 times its expected value, and for element Ny this ratio was 1.99. The 
amounts of elements n,, and n,, were close to their expected values. The values were 1.22 times and 1.21 times 
the expected values, respectively. Portugal, Switzerland, and Germany decreased three types of relative errors. In 
Portugal, for example, the values of elements N,N,N, and n,, were 1.71, 1.17, 1.16, and 1.89 times the expected 
values, respectively. The decrease in the number of students in elements n Nor N53, aNd 1, especially in n,,, re- 
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Table 2 
Four Types of Equality in Quality Science Education Systems 


Disadvantaged Advantaged 
: students students Reguiar students 
Tiss Countries/ 
yp economics R’% N . ‘ : 
‘i Relative Relative Relative 
1 error % 2 error % 3 error % 
France 20.1 6176 1417 26.58 1419 26.48 2461 59.39 
Belgium 20.0 8312 1964 24.39 1930 25.69 3228 55.34 
Czech Republic 16.9 6911 1627 24.67 1637 24.20 2665 54.25 
Low-equality 
Germany 18.6 4656 1143 21.44 1145 21.31 1740 49.48 
Switzerland 16.3 5705 1428 19.91 1400 21.47 2092 46.68 
Portugal 15.9 5632 1414 19.66 1395 20.74 2086 48.15 
Australia 10.0 12813 3304 17.48 3284 17.98 4422 38.06 
Equal-start 
Norway 8.9 5612 1498 14.58 1454 17.09 1878 33.86 
Estonia 7.2 5202 1376 15.36 1377 15.29 1697 30.49 
Latvia 8.4 5187 1388 14.37 1374 15.23 1683 29.79 
; Equal Canada 6.4 21490 5780 13.93 5665 15.64 7001 30.31 
improvement 
Korea 8.0 6626 1779 14.08 1763 14.86 2173 31.18 
Japan 77 6055 1650 12.80 1619 14.44 1924 27.10 
Hong Kong (China) 57 5839 1650 9.58 1621 11.61 1836 25.77 
Egalitarian 
Macao (China) 1.6 3766 1132 3.81 1117 4.92 1043 10.78 


Note. R-squared is the explanatory power of ESCS on students’ science scores in that country or economy, as documented in the 
PISA 2018 results (see Table II.B1.2.4, OECD, 2020a). N is the total number of subjects who took the PISA 2018 test without miss- 
ing ESCS values. 


Elements n,, and n,, were in the low-score area, and n,, and n,, were in the high-score area. Therefore, a 
reduction in the relative errors for disadvantaged and advantaged students would usually reduce the number of 
students on the main diagonal and thus reduce the relative error for the regular student. However, an anomaly was 
found when comparing the Swiss and Portuguese samples. Portugal had lower relative errors for disadvantaged 
and advantaged students but a higher rather than lower relative error for regular students. That was due to the 
migration of students from element n,, to n,,. In the Portuguese sample, the value of n,, was only .989 times the 
expected value (in general, the value of this element was significantly higher than the expected value). 
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Three Types of Equality in Equitable Science Education Systems 


The equal-start system was named because the education system started to help disadvantaged students instead 
of just chanting slogans. The significant reduction in the relative error for the regular student indicated a rapid decline 
in the numbers in elements n,, and n,,. In Australia, for example, the values for n,, and n,, were 1.61 and 1.64 times 
the expected values, respectively. 

The equal-improvement system was named because it further reduced the achievement gap between disad- 
vantaged and advantaged students compared to the equal-start system. There were two ways to improve equality in 
the education system. Estonia and Canada each represented one. In Estonia, disadvantaged students moved to the 
high-score area, and advantaged students moved to the low-score zone in a balanced way. This approach led to a minor 
difference in the relative error for disadvantaged and advantaged students. In Canada, the number of disadvantaged 
students moving to the high-score area was higher than the number of advantaged students moving to the low-score 
area. This practice resulted in a lower relative error for disadvantaged students than for advantaged students. 

The egalitarian system got its name because it achieved a high degree of educational equality that would not have 
been possible without the emphasis on equality. Such education systems paid much more attention to the educational 
benefits of disadvantaged students. In the Hong Kong (China) sample, element n,, was only 1.38 times its expected 
value. It was further reduced to 1.18 times in the Macao (China) sample. It led to a rapid increase in the number of 
disadvantaged students achieving science achievement levels that exceeded their family capital levels. However, there 
was also a further increase in advantaged students migrating from high to low-score areas in this education system. 


The Probability of Academic Success in Science for Students with Different Family Capital 


Table 3 shows the information on successful students in the science domain. It focused only on the four ele- 
ments of the fourth column of the achievement matrix. Therefore, the analysis of equality in science education 
from Table 3 is a partial and one-sided perspective. But it can complement the information in Table 2. The transition 
from alow toa high equality type demonstrated a sustained increase in the probability of disadvantaged students’ 
success in science. In the low-equality type education system, the most advantaged students were five or more 
times more likely to succeed in science than the most disadvantaged students. In the Czech Republic, for example, 
only 6.48% of the most disadvantaged students scored in the upper quartile region, while nearly half of the most 
advantaged students’ scores were in the upper quartile region. Compared to the low-equality education system, 
the equal-start education system had an increase in P, and a decrease in P.,. It reduced the gap in academic success 
between the most disadvantaged and the most advantaged groups. In the equal-improvement system, the P, value 
increased consistently and raised equality compared to the equal-start system. Japan was special. Although its P, 
value was comparable to the equal-start system, its P, value was higher than the rest of the sample in the equal- 
improvement education system. In the equal-improvement type system, there were approximately six advantaged 
students and three disadvantaged students for every nine successful students. In an egalitarian system, P, and P, 
increased while P, decreased. The gap between P,, P.,, P,, and P, was further reduced. 


Table 3 
The Conditional Probability of Success in Science for Secondary School Students with Different Family Capital across Countries/ 
Economics 


: Conditional probabilities Ratios of conditional probabilities 
Countries/ 
Types , 

economics 
P41 P2 P3 P4 P4/P1 P3/P1 P2/P1 
Czech Republic .0648 .1700 .2888 4756 7.340 4.457 2.623 
France .0704 .1524 2796 A976 7.068 3.972 2.165 
Belgium .0844 1676 2776 4708 5.578 3.289 1.986 

Low-equality 

Germany .0860 .1856 2724 4560 5.302 3.167 2.158 
Switzerland .0916 1872 .2700 4508 4.921 2.948 2.044 
Portugal .0936 .1876 2472 A716 5.038 2.641 2.004 


Ass https://doi.org/10.33225/jbse/23.22.439 


Journal of Baltic Science Education, Vol. 22, No. 3, 2023 


ISSN 1648-3898 /Print/ TYPES OF QUALITY AND EQUAL SCIENCE EDUCATION SYSTEMS AND THE DEVELOPMENT OF 
ISSN 2538-7138 soniine/ Ow a Sayeeda 
Soins Conditional probabilities Ratios of conditional probabilities 
Types ountries 
economics 
P1 P2 P3 P4 P4/P1 P3/P1 P2/P1 
Australia 1240 1832 2820 4108 3.313 2.274 1.477 
Equal-start 
Norway 1160 .2016 2944 .3876 3.341 2.538 1.738 
Japan 1104 .2160 .3000 3732 3.380 2.717 1.957 
Estonia 1392 1800 2736 4068 2.922 1.966 1.293 
; eal Latvia 1372 1868 2892 3864 2.816 2.108 1.362 
improvement 
Canada 1368 1916 2936 3779 2.762 2.146 1.401 
Korea 1372 1976 2776 .3876 2.825 2.023 1.440 
Hong Kong (China) 1632 2144 2376 3843 2.355 1.456 1.313 
Egalitarian 
Macao (China) 1912 .2476 2348 3260 1.705 1.228 1.295 


Note. P, = P [score = 4| ESCS = 1], P, = P [score = 4 | ESCS = 2], P, = P [score = 4 | ESCS = 3], P, = P [score = 4 | ESCS = 4]. 
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Discussion 


The relative error is an indicator of equality in education. The coefficient of determination deals with the as- 
sociation between variables (Nachbauer & Kyriakides, 2020; OECD, 2019). In contrast, the relative error is an indica- 
tor related to the person. A decrease in the relative error for disadvantaged students meant that more students 
migrated from lower to higher-scoring areas. A reduction in relative error for advantaged students implied the 
Opposite migration process. Thus, a decrease in the relative errors for advantaged and disadvantaged students 
indicated the mechanism by which the achievement gap between the two groups of students was reduced. When 
comparing the Czech Republic with Germany, the coefficient of determination showed that educational inequality 
was higher in Germany (OECD, 2020a). However, the relative errors for disadvantaged and advantaged students 
showed that educational inequality was higher in the Czech Republic than in Germany. 

The application of relative error is very flexible. For example, society may not consider that the migration of 
the most disadvantaged students to the n,, also represents an increase in educational equality. The score in the 
region between the lower quartile and the median may not help students’ future position in the labor market. 
In other words, students in elements n,, and n,, will be engaged in the same work. Only when students migrate 
to n,, and n,, can education equality be improved. At this time, the relative error for disadvantaged students can 
be calculated based on the four elements of Nigr M4 M44 and n,,. However, this study did not favor omitting n> 
to calculate the relative error for disadvantaged students. This approach was one-sided because it only looked 
at the issue of educational equality from the perspective of academic success. It deviated from the core concern 
of equality, ‘person.’ It regarded the struggle and efforts of the most disadvantaged students in element n,, as 
insignificant. On the contrary, this study considered each student in elements NM, and n,, as having the same 
significance and value for educational equality. 

Conditional probability is a complementary indicator of equality in education. If equality in education must be 
assessed from the perspective of academic success, the indicator conditional probability can be used. In addition 
to being used as an indicator of education equality, it can also be used as an indicator of the national development 
index. The P, values for the low-equality type systems were all less than .1, and the P, were all greater than .45. By 
2024-2025 (six to seven years following 2018), only one in every eleven to fifteen new STEM workers will come from 
the most disadvantaged families. In an egalitarian education system, the situation will be much better. In Macao 
(China), between 2024 and 2025, for every five new STEM workers, approximately two will come from the most 
advantaged families and about one from each of the other three family capital levels. If disadvantaged students 
see no hope of achieving upward career mobility through science education, it can lead to many of them slacking 
off (McNutt, 2020; Next Generation Science Standards, 2013, p. 29). That would be detrimental to the country's 
high-tech human resource building in a globalized community. 

Countries or economies develop educational equity in the competition between equality and quality. Gener- 
ally, in the contest between them, the elitist tradition of education makes the nation pay much more attention to 
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quality than equality (Pitman & Vidovich, 2012). This practice has resulted in a large academic gap between disad- 
vantaged and advantaged students. In the control group, there was a close connection between students’ scores 
and their family capital. Students were gathered in the four elements on the diagonal of the achievement matrix. 
The most disadvantaged students were less likely to achieve academic achievement grades that exceeded their 
family capital levels. Rawls (2005) has argued that quality alone does not ensure equity in education (p. 71). He 
has proposed that education should guarantee the benefits of the most disadvantaged students (Rawls, 2005, p. 
80). Estonian secondary education balances equality and quality (Lees, 2016; Tucker, 2015). It has enabled Estonia 
to achieve equity in science education. A large number of students moved out of the elements on the diagonal, 
making the relative error for regular students in the Estonian sample about 30% smaller than that in France. The 
experimental group sample showed that the countries or economies valued equality in education. That was due 
to the full consideration of the political value of equality in promoting social cohesion. 

In the experimental group, the disadvantaged students’ performance improved while the gap between them 
and the advantaged students was reduced. It may be argued that this was unfair to the advantaged students. How- 
ever, according to Rawls’ principle of difference, advantaged students can benefit from improving disadvantaged 
students’ performance. That is because the disadvantaged students’ increased achievements prompt advantaged 
students to adjust their academic expectations and set higher achievement goals (Rawls, 2005, pp. 76-80). Higher 
goals motivated advantaged students to strive for better achievement. In this way, the academic competition be- 
tween disadvantaged and advantaged students ensured equality and promoted quality at a higher level. Students 
in Macao (China), Estonia, Japan, and Korea achieved outstanding science performances in an equal climate. They 
were examples of “increased equality benefiting both disadvantaged and advantaged students.” 


Conclusions and Implications 


The key measure to improve educational equality was to ensure the educational benefits of the most disadvan- 
taged students. In the low-equality education system, the number of students in element n,, reached approximately 
two times its expected value. In the experimental group, the value gradually decreased to about 1.2 from the equal- 
start type to the egalitarian type system. In the low-equality system, the number of students in element n,, was 
approximately 1.2 times its expected value. There is not much room for that figure to fall. Thus, Rawls’ profound gaze 
had penetrated the crux of the problem of equality in education half a century earlier. When educational practice 
aims to ensure the educational benefits of the most disadvantaged students, it can promote educational equality. 

The findings exposed the path toward developing equality in a quality science education system. Moving from 
a low-equality type to an equal-start type system required a substantial reduction in the relative error for the regular 
student. It implied a rapid decrease in the number of students in elements n,, and n,, on the main diagonal. This 
measure mitigated the consistency between family capital and achievement. The path to high equality in an equitable 
science education system was paved by disadvantaged students continuously narrowing the academic achievement 
gap with advantaged students. The advantaged students previously held a great achievement advantage over the 
disadvantaged students. This great advantage is unfair. It is an anomaly that derives from the family capital. Impro- 
ving equality meant substantially improving the academic performance of disadvantaged students without causing 
a regression in the performance of advantaged students. That narrowed the huge advantage held by advantaged 
students without compromising their educational gains. Macao (China), Estonia, Japan, and Korea were examples of 
such education systems. 

There were three types of equitable science education systems, which were the objective reference for science 
education across countries. The relative errors for disadvantaged and advantaged students in the equal-start system 
were between 16% and 19%. The two kinds of relative errors between 12% and 16% were equal-improvement type, 
and those below 12% were egalitarian type. In addition, in quality but low-equality education systems, these rela- 
tive errors were more than 19%. This study did not analyze countries or economies with a determination coefficient 
between 10% and 15% in the PISA test. These education systems can be classified as low-equality or equal-start type 
based on their relative errors. 

The study has several implications. First, new indicators have been constructed to assess equality in education. 
Relative errors make a more precise distinction between levels of equality. Second, since the least advantaged stu- 
dents gather in the lowest-scoring areas, they need external help more urgently than any other group. The education 
benefits of the most disadvantaged are well guaranteed in quality and equal science education systems. It provides 
evidence that Rawls’ difference principle has guided the practice of developing equality in education systems. Third, 
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the reward is substantial for the nation to devote additional financial resources to developing educational equality. 
Increased equality in education will lead to the emergence of many people with high academic performance among 
disadvantaged groups. It broadens the source of talent in the field of STEM. The analytical techniques of this study can 
be applied to a wide range of educational equality issues. It has been suspected that students with lower academic 
performance receive less teacher guidance. That is an issue of equality in the educational process. It can be examined 
using student achievement and perceived teacher feedback as the row and column variables of the matrix, respectively. 


Limitations and Future Research 


The achievement matrix is not applicable for nominal variables. For example, the achievement matrix analysis 
technique is inappropriate when the researcher is interested in gender equality in science education. The results 
described three types of equitable science education. It can guide the direction of science education reform across 
countries but cannot provide specific strategies or educational policy details. Future research should further explore 
the impact of curriculum and educational policies on the development of science education. As can be seen from 
Tables 2 and 3, Latvia and Estonia share similar educational equality. However, Estonia's science performance exceeded 
Latvia's by 43 points. The difference in the quality of science education between the two countries may be due to the 
science curriculum and examination standards or to the training and preparation of science teachers. The real reasons 
need to be explored. The findings provided the path from a low-equality system to an equal system and the evolution 
of equality in equitable science education systems. However, this study cannot answer how to transition from a low- 
quality to an equity-type education system. Follow-up studies should select appropriate samples for data analysis 
and domestic education policy analysis. A 4 x 4 achievement matrix was used in this study. The most disadvantaged 
students were those with family capital in the bottom quartile. Other studies that define the most disadvantaged 
students as having less than 12.5% family capital can employ an 8 x 8 achievement matrix. 
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Introduction 
Research Problem 


The current education system in Spain (Organic Law, 2020) contem- 
plates European recommendations on key skills for lifelong learning as a 
response to the demand for active learning in the training of responsible 
citizens. Its incorporation into the curriculum is therefore essential, being 
necessary to emphasise those aspects of learning orientated towards ap- 
plying the acquired knowledge in response to new educational needs from 
an integrated approach. 

In the case of science education, these needs are related to the promo- 
tion of the scientific-technological literacy of citizens and the development 
of scientific knowledge, skills and attitudes in different contexts, responding 
to everyday life situations that arise (Gilbert et al., 2011). It is an approach in 
which 21*-century learning scenarios require a treatment of science teaching 
that is able to link central disciplinary ideas with scientific and engineering 
practices, transversal concepts and all types of processes (Morrell et al., 2020). 

This educational framework requires competent professionals, de- 
manding that teachers have appropriate training in order for them to act 
in accordance with the same model they want to teach (Lupidn-Cobos et 
al., 2023; Lupidn-Cobos & Gallego-Garcia, 2017). Thus, and with regards to 
teachers, new challenges are associated with the skills model that is added 
to the personal needs of teachers themselves, who, as reflexive professionals, 
must question ways of understanding how to teach science (Confederacién 
de Sociedades Cientificas de Espana [COSCE], 2011; European Commission, 
2015; National Research Council [NRC], 2012). 

Within this framework STE(A)M education arises, with benefits associated 
with the integration of the knowledge and promotion of scientific-techno- 
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logical identities (Grimalt et al., 2021), technological-scientific literacy, social justice and sustainable development 
(Doménech, 2018) through the combining of technology and engineering via problems that are socially relevant 
for students (Ortiz-Revilla et al., 2020). STE(A)M in the Spanish educational context today promises innovation and 
preparation for the future. The term is becoming familiar not just to teachers and educators but also students and 
their families through a range of programmes, initiatives, and resources developed both inside and outside school 
(Castro-Rodriguez & Montoro, 2021; Couso & Simarro, 2020). 

Its curricular inclusion in the Spanish education system has been developed thanks to the contribution, on 
the one hand, of the International Education Agency (UNESCO, 2019), which underlines the value of STE(A)M 
education for achieving the skills-based development the 215-century demands and, on the other, of the Spanish 
Strategy for Science, Technology and Innovation 2021-2027 (Ministerio de Ciencia e Innovacién [MICINN], 2021), 
which proposes tackling STE(A)M education from early stages, with the aim of reaching quality science through 
inclusion and diversity. 

Although increasingly diverse studies are beginning to show guidelines and offer opportunities for the imple- 
mentation of STE(A)M education in the Early Childhood, Primary and Secondary Education stages (Aguilera, 2022) 
under the umbrella of the current education law (Jefatura del Estado, 2020), the correct implementation of it is, 
however, both ambitious and complex, leading to teaching difficulties or tensions related to its conceptualisation 
or the training teachers themselves have (Toma y Garcia-Carmona, 2021). These teaching tensions (Doménech, 
2018; Pérez-Torres et al., 2021), which ultimately suppose a challenge to the transfer of STE(A)M education to sci- 
ence classrooms, are widely described in the recent literature related mainly to issues of curriculum organization 
and teacher training (Affouneh et al., 2020, El Nagdi et al., 2018; Margot & Kettler, 2019) as well as to personal 
characteristics and systemic factors, internal and external of institutions, being necessary to identify and evaluate 
them in favour of developing effective proposals. 

In this regard, Professional Development (PD) programs focus on developing attitudes, skills and knowledge 
that are needed to provide useful training approaches to address the necessary teaching domains, which involve 
the selection and combination of curricular elements, teaching approaches, and the integration of contents and 
processes that participate in the learning STE(A)M situation. 

Combining knowledge of what is already known about education in Science, Engineering or Mathematics 
with the addition of the potential and limits of disciplinary integration that STE(A)M multidisciplinarity entails and 
comfort with using technology in creative ways to increase integration of STE(A)M in classrooms (Gamse et al., 2017) 
requires a significant professional effort. Specifically, closely related to design, a high-level professional competence 
for teachers (De Vries, 2020). Hence, the treatment of STE(A)M education challenges requires support for progressive 
teacher professional development when implementing its principles in the classroom (De Meester et al., 2021). 

On the other hand, the systemic nature of STE(A)M education implies different levels of internal teacher col- 
laboration within school teams, all of them mobilizers of teacher exchanges, such as the availability of training 
material, jointly analyzing well-designed examples of STE(A)M curricular activities and providing the time to built 
the processes by the student-centred approach, letting students be active learners (Gasiewski et al., 2012). 

PD is the key to helping teachers through the transformation process (Shahali et al., 2015). For this reason, it 
is essential to select, adapt and sequence proposals for it. Even more when linking them to the “decision-maker” 
and “teacher as designer” models. (De Vries, 2020). 


Research Aim and Research Questions 


Different studies have shown the importance of professional development programs in STE(A)M teacher 
training, both with practising teachers (Yildirim et al., 2022) and with teachers in initial training (Sahali et al., 2015). 

The possible benefits of an integrated approach depend largely on how it is carried out, recognising teach- 
ers as a catalyst for the process (Margot & Kettler, 2019) and is limited by the specific and methodological training 
used. Related to the methodological aspects, Thibaut et al. (2018) concluded that collaborative work, interdiscipli- 
narity, inquiry-based learning, problem- solving and design-based learning were the most used methodologies. 
The positive effect of inquiry-based teaching on student cognitive and attitudinal outcomes (Marshall & Alston, 
2014), who extend their dedication to a regular interest in science, provide us with a valuable and promising tool 
to approach STE(A)M relationships in everyday school life, including problem-based, inquiry-based approaches, 
and design-based pedagogies or the assessment of multiple STE(A)M learning outcomes, among others ina series 
of defining characteristics associated with the design and implementation of STE(A)M units (Toma & Greca, 2018). 
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However, it is surprising the small number of studies which analyse the enactment of integrated STE(A)M proj- 
ects (McLure et al., 2022), particularly when teachers have indicated that they need more direction and support on 
how to effectively integrate learning areas (Margot & Kettler, 2019). In addition, Roehrig et al. (2021) have identified 
that making explicit connections between the content of the targeted disciplines is one of the essential factors 
for effective STE(A)M projects. 

To put integrated STE(A)M projects into practice (McLure et al., 2022; Van Driel et al., 2012) implies teachers 
need design skills to elaborate classroom activities based on inquiry for STE(A)M education (Affouneh et al., 2020), 
combining conceptual learning with professional practices (McLure et al., 2022: Roehrig et al., 2021), taking into 
account the attitudes, beliefs, knowledge and teaching skills and also considering the existence of internal condi- 
tioning factors, along with interactions between these, their social environment and the education system itself 
(Affouneh et al., 2020; Lupidn-Cobos et al., 2022). 

The present study has focused on the Spanish context and aims to answer the following research questions: 

e¢ What level of teaching inquiry do Primary Education teachers show when implementing inquiry-based 
STE(A)M projects, and how do they evolve with practice? 

e What is their perception, both at the level of implementation and of the established collaborative 
framework, as regards the advantages and difficulties encountered? 


Research Methodology 
Study Context & Design 


The IndagaSTEAM Escuela project (Lupidn-Cobos et al., 2023) has arisen to contribute to the need to promote 
the transfer of the results of science education research to educational institutions with the purpose of moving Sci- 
ence and Technology closer to schools and developing critical thought amongst their students (Millar & Osborne, 
2009; Perines, 2018). The project is thus constructed as a useful training scenario for applying the objectives of 
STE(A)M education, combining the strategies of inquiry and contextualisation to do so as an integrative way to 
incorporate basic knowledge from several disciplines through active students’ participation and a mentoring of 
the training progression with design and implementation of own projects (Morrell et al., 2020). 

The IndagaSTEAM Escuela project is an initiative promoted by public institutions made by the regional edu- 
cation administration of Andalusia via cooperation between colleges and universities (Lupidn-Cobos et al., 2023). 
The project functions through interventions in which schools voluntarily participate in innovation and research 
processes orientated towards demands in improvements that involve their educational communities. The project 
has been developed for three years from a vision of STE(A)M education as a framework for linking knowledge 
and understanding via the interrelated threads of the specific content of the areas involved (Science, Technology, 
Engineering, Mathematics, Arts). 

Employing situations relevant to students through the use of inquiry-based learning and the use of relevant 
contexts and socio-scientific issues as learning scenarios encourages students’ interest and involvement and fa- 
vours the understanding of the nature of science (Lupién-Cobos et al., 2022). The projects boost their capacity for 
making informed decisions, develop evidence-based arguments in relation to local, national and global problems, 
and put their key skills into practice by working cooperatively. The teaching staff involved in the scientific educa- 
tion research carry out the role of project trainers, orientating a training modernisation and guidance project for 
school teachers in the process of professional development for them that implies both theoretical foundation and 
ideas for practical implementation. 

Thus, this research considers the beliefs of teachers about STE(A)M Education and the need for specific training 
for its implementation (Herro & Quigley, 2017) when contemplating the training objectives of the IndagaSTEAM 
Escuela project, which also articulated an expectation of professional practice for the transfer of innovative teach- 
ing approaches through the following aspects: 

¢ Developing capacity for teaching inquiry, favouring its reflection processes and ability to manage 
transformative learning environments. 

e¢ Promoting a correct integration of the different elements that have an impact on the development 
of these innovative teaching approaches (curriculum, school organization, professional development 
and didactic material, among others). 
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¢ Generating teaching-learning situations appropriate for conceptualising the contextualized teaching 
model, inquiry and STE(A)M undertaken, clarifying educational purposes and relationships in their 
treatment. 
¢ Supporting teaching staff in their teaching capacity for selecting and designing projects that permit the 
formation of coherent and solid proposals with the intention of the proposed model, their identification 
and design. Thus, the design can be considered a form of complex problem-solving involving interdis- 
ciplinary thinking, handling multiple decisions and unanticipated problems, inquiry, collaboration in 
multidisciplinary teams, and a scientific basis from which to start (de Vries, 2020; De Meester et al., 2021). 
e Systematically assessing the impact of actions and providing evidence/research evidence to guide 
improvement. 
The counselling offered by the PD programme provided training support to promote an interrelated un- 
derstanding of the processes involved, following a training progression plan (Figure 1), in three phases: Initial (I), 
Development (D) and Final (F) 


Figure 1 
Phases of the Training Programme 


INTRODUCTION TO l TRAINING & D DESIGN, F EXCHANGE OF 
DRIVEN TEACHING = PREPARING THE > IMPLEMENTATION > VIEWS & 
APPROACHES DESIGN & ASSESSMENT OPINIONS 


The training complexity involved in dealing with real-world problems and the very nature of the STE(A)M 
treatment (Couso & Simarro, 2020) to design projects was addressed using different levels of curricular integration 
(Pérez-Torres et al., 2021) and the participation of scientific inquiry processes in the activities to reflect conclusions or 
improve products. Thus, its educational components (context, problem and final product) were established to pro- 
mote student competencies (Domenech-Casal, 2018) and the strategies for its teaching design and implementation. 

The epistemological, scientific and educational scenario, which involved its planning and elaboration by 
the teacher and its established relationships, represents a starting line for research on teacher training in STE(A) 
M education. 

The project was developed from 2019 to 2022 in a public Early Childhood and Primary Education Centre (CEIP) 
in Andalusia, in the framework of the collaborative education convention subscribed to between the Department of 
Education and Sport of the Regional Government of Andalusia (CEJA, 2017) and the public universities of Andalusia, 
in the sphere of educational innovation and research. A total of eight Primary Education teachers participated in 
a voluntary professional development initiative in order to tackle the challenge of introducing STE(A)M education 
as an approach for the improvement of the scientific-technological learning of their students. 

At the heart of the IndagaSTEAM Escuela project, the teaching proposals designed by the researchers from 
the University of Malaga adhered to a model for the transfer between training and teaching practice, orientating 
themselves towards conceptualising and sharing with teachers their vision of the STE(A)M approach, clarifying 
both the educational proposals and goals (scientific-technological literacy of students, integration of the gender 
perspective, emerging technologies) of the teachers and selected perspectives and methodologies, with a strong 
basis on what it is already known about education in Sciences, Engineering and Mathematics, with the addition 
of the potential and limits of disciplinary integration in curricular treatments. Furthermore, there are internal and 
external systemic factors, associated with the need for collaborative scenarios in the centre for their putting into 
practice. To address this epistemological and behavioural structure, there was a contemplation of frameworks 
throughout the three academic years of the development of the project, with teaching progressions considered 
in relation to dimensions such as dependence/autonomy, isolation/coverage and task decontextualisation/con- 
textualisation (Monereo, 2010). 

Teaching resources were used as exemplifications in the initial training period (corresponding to the 2019/2020 
academic year) and designed ad hoc, including teaching guides and class materials, among others, which address 
specific topics relating to physical-chemical processes. 
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Appendix 1 includes a support file for teachers to prepare the educational guide for their school project. In 
it, and associated with promoting their competence in teaching design and planning, they identify the selected 
teaching approaches, the curricular elements that their proposal would contemplate, and the implementation and 
processes of metacognition and reflection that the intervention can offer. 

Subsequently, throughout the second and third years, teachers created their STE(A)M proposals with the 
mentoring of the researchers, preferably focusing their projects on important topics aligned with the Sustainable 
Development Goals (SDG) of the 2030 Agenda (Schleicher, 2018), such as projects relating to health and wellbeing, 
sustainable cities and communities, and climate action, on the whole. 

With the experience accumulated following the development of the IndagaSTEAM Escuela project over the 
last three academic years and considering the mutual interaction between what teachers do and what they think 
(Ariza et al., 2021; Garcia et al., 2019), this research analyzes the challenges and opportunities of the training received 
manifested in teachers’ beliefs about STE(A)M education and in their teacher self-perception of their teaching in 
relation to the incorporation of integrating STE(A)M proposals. 


Participants 


The longitudinal case study presented here focuses on the involvement of two of the teachers from the third 
cycle of Primary who took part in the training programme and completed the entire process over the three years of 
the duration of the project in addition to using all of the research instruments set out in its development. This was 
a convenience sample in which the teachers participated in the research on a voluntary basis. They authorised the 
researchers to observe their classes, interview them, and use the reports and other teaching materials they designed 
during the training programme for the purposes of the research. Of course, these teachers are not representative 
of all primary school science teachers, and this might be considered a limitation of the study; however, they are 
representative of a sector of the teaching staff who show interest in their training and in taking on the challenges 
of reforming school science curricula (Organic Law, 2020). 

Teacher 1 (from now on, T1) had 33 years of teaching experience after obtaining the academic Degree in 
Primary Education. T1 had carried out previously, and throughout its long history, numerous educational innova- 
tion experiences (the majority focused on the use of new technologies). This teacher also had previous experience 
collaborating on research projects at the University of Malaga and assumed responsibility for the centre’s school 
management. For her part, teacher 2 (from now on, T2) had 12 years of teaching experience and the same academic 
degree as T1. However, unlike T1, T2 only had one experience in educational innovation and research projects. 


Instrument 


The implementation of the IndagaSTEAM Escuela project focused on teaching practices in terms of the STEAM 
community, STEAM integration, and STEAM assessment using the approach of the collaborative training model for 
professional development in schools, in which the teaching accompaniment began in the 2019/2020 academic 
year, with an introduction to the driving approaches of the project (inquiry, contextualisation and STE(A)M) and 
a mentoring of the training progression in the successive academic years 2020/2021 and 2021/2022, with design 
and implementation of our own projects. 

Different research instruments were applied throughout this training process. Figure 2 shows a summary of 
the data collection timeline. 
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Figure 2 


Chronology of IndagaSTEAM Escuela Project and Data Collection Instruments 


2019/2020 


INTRODUCTION TO THE E 
PROJECT AND SPECIFIC 1° YEAR OWN DESIGN 2°t YEAR OWN DESIGN 


TRAINING PRENTORING) (MENTORING) 


iaas INITIAL INQUIRY FINAL INQUIRY 
TEACHING LEVEL TEACHING LEVEL 
~- Planning 


- IndagaSTEAM focus exchange Teacher observation rubric Teacher observation rubric 
- Examples and analysis of experiences (adapted from Marshall et al., 2009) (adapted from Marshall et al., 2009) 


- Elaboration of support materiales 
Implementation INITIAL PERCEPTIONS FINAL PERCEPTIONS 
Interviews SWOT analysis Interviews SWOT analysis 


- Carrying out preparatory session 
PROJECTS INITIAL ASSESSMENT PROJECT FINAL ASSESSMENT 


- Detection improvement opportunities 
Likert-type questionnaire Likert-type questionnaire 


- Implementation of guided projects 
- Collection of data and evidence 


- Analysis and assessment of 
intervention results 

- Identification of potentialities and 
improvements 


Thus, the researchers and authors of this article who acted as observers during the workshops applied a ru- 
bric adapted from the EQUIP instrument by Marshall et al., (2009) (Appendix 2), which recorded the evolution of 
the level of teaching inquiry. Using a 4-point Likert scale (level 1: pre-inquiry; level 2: inquiry under development; 
level 3: competent inquiry and level 4: exemplary inquiry), the rubric covers aspects referring to the necessary skills 
and competencies for implementing this educational approach in the Primary Education classroom, such as the 
teaching role (in its action as a facilitator or guide of the teaching-learning process), communication strategies (as 
regards capacity to foster dialogue more or less guided by students) and class interactions (in relation to capacity 
to afford continuity to questions and answers arising during inquiry through argumentation), among others. This 
rubric was applied in the academic years the proposals were implemented, 2020/2021 and 2021/2022. 

Additionally, semi-structured interviews were carried out, designed ad-hoc, with open-ended questions. A 
SWOT (Strengths, Weaknesses, Opportunities and Threats) matrix was applied for the design of the questions, an 
analytical technique with business origins that is simple to use and provides an excellent tool for evaluating reality 
and making decisions. Thus, it was possible to group the interview questions, which were organised randomly, 
into four categories to analyse external aspects (Weaknesses, Strengths) and internal ones (Threats, Opportuni- 
ties) (Figure 3). 
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Figure 3 
SWOT Matrix Interview Structure 
STRENGTHS WEAKNESSES 
What specific aspects can projects help you in What teaching shortcomings do you find in 
your science classes? carrying out the projects? 


: ; What external difficulties are you encounter- 
What expectations do you have for the imple- ne ; ; : 
; : ; ing in the design and implementation of the 
mentation and evaluation of the projects? 


projects? 
OPPORTUNITIES THREATS 


Finally, and with the aim of offering a more detailed view of the context of the intervention, two question- 
naires with a 5-point Likert scale were provided, orientated towards discovering both the valuation of the projects 
designed and implemented by the participating teachers (questionnaire 1) and the working methodology estab- 
lished via the collaboration agreement between the Early Childhood and Primary Education Centre involved and 
the University of Malaga (questionnaire 2). Tables 1 and 2 show the statements formulated in the first (S$1-S8) and 
second (S9-S15) questionnaires, respectively, and were grouped into categories aligned with the challenges and 
opportunities described by Margot and Kettler (2019) related to teaching collaboration, curriculum, institutional 
support, methodology, professional development, teaching projection and student concerns. 


Table 1 
Questionnaire 1 


Statements Category* 


The strategies used in this project have stimulated the effort of my students and promoted 


= their ability to learn by themselves and with others. SRaCe COGS 
The contents that have been worked on in this project are interesting for applying the Primary : 

$2 . ; ; : Curriculum 
Education curriculum to issues of interest to my students. 

$3 The realization of this project has meant an improvement in my students’ learning. Student concerns 

SA Carrying out this project has meant an improvement in the coexistence relations of my Methodolo 
students with their families, the immediate environment, the teachers and our school. gy 

$5 Working on science through the strategies of this project represents an important methodo- Methodology 
logical change in my science class. 
The design of the materials used in this project has allowed me to create useful teaching 

$6 ee P ; Student concerns 
situations for my students’ learning. 

S7 The implementation of this project has helped me reflect on my teaching practice. Professional development 


The scientific projects set aside with these methodological guidelines as support material 
$8 have helped me to connect the aspects of the curriculum, and it is useful to plan the evalu- Curriculum 
ation of the students. 


*Following Margot & Kettler’s (2019) categorization 
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Table 2 
Questionnaire 2 
Statements Category* 
$9 Participating in this collaboration with the university has met my initial expectations. Teaching collaboration 


The relationships established between the participants in the project have facilitated putting into 


$10 F : F ae Teaching collaboration 
practice active teaching strategies in my class. 
Sit The resources and strategies provided by the project have been of practical use for my classes. _ Professional development 
4 Using questions about specific situations in students’ daily life is a practice that helps students ’ 
$12 ; ; : Curriculum 
improve their knowledge about the processes that occur in them. 
$13 The collaboration has been adjusted to the training levels of the participating teachers. Professional development 
$14 The collaboration has promoted spaces for communication and reflection Professional development 
$15 Overall assessment of your expectations achieved with the project. Teaching projection 


*Following Margot & Kettler’s (2019) categorization 


Data Analysis 


The interviews were registered on paper and then analysed by the researchers who authored this article, 
experts in Experimental Science Teaching and teacher training. They were analysed qualitatively using coding 
techniques carried out with the software program Atlas.ti (versidn9.0.20.0) (www.atlatsti.com). The researchers 
and co-authors of this work read the registries a number of times and identified common aspects within each of 
the established categories. 

For the Likert-type questionnaires, a quantitative descriptive analysis was carried out with the program RStu- 
dio (version 1.3.1093), comparing the score averages that teachers awarded for each question, initially during the 
20-21 academic year and on finalising the involvement during the 21-22 academic year. As the questions from 
both questionnaires were formulated in the affirmative, it has been considered only those that were found in the 
category “agree” and “completely agree” categories to be affirmative responses. 


Research Results 
Teaching Inquiry Profile 


The results obtained regarding the teaching inquiry profile in the 2020/2021 academic year (Table 3) show 
an initial level of inquiry in development (with an average score of 1.96 out of 4.00 points). As favourable aspects, 
those related to the dynamic of the debates established particularly stand out, in which the teachers managed to 
successfully involve their students through open questions (average of 2.50), or their teaching role in the inquiry 
processes, in which they occasionally acted as facilitators (average of 2.25). 


Table 3 
Results of the Initial Level of Teaching Inquiry 


Assessed actions M (SD) Level of teaching inquiry 
Educational strategies 1.75 (0.71) Developing inquiry 
Order of instruction 2.00 (0.00) Developing inquiry 
Teacher role 2.25 (0.71) Developing inquiry 
Student role 1.50 (0.71) Developing inquiry 
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Assessed actions M (SD) Level of teaching inquiry 
Questioning ecology 2.50 (0.71) Proficient inquiry 
Communication pattern 1.75 (0.00) Developing inquiry 
Classroom interactions 2.00 (1.41) Developing inquiry 
Global 1.96 (0.61) Developing inquiry 


Notwithstanding, the panorama observed throughout the 2021/2022 academic year was considerably more 
satisfactory (table 4). In the progression of the training framework of the project, it is in this last year that the teach- 
ers took on the role of “teacher as a designer’, and from this, advances are observed in specific teaching domains, 
such as the level of competent inquiry, with the evolution of both teachers standing out in aspects related to their 
role of facilitators and not just drivers of the process, and in the contextualisation of tasks, strategies and the order 
of instruction assumed, which now favour environments aimed at student autonomy for tackling investigations 
and exploring their own ideas. 


Table 4 
Results of the Final Level of Teaching Inquiry 


Assessed actions M (SD) Level of teaching inquiry 
Educational strategies 3.50 (0.71) Exemplary inquiry 
Order of instruction 3.00 (1.41) Proficient inquiry 
Teacher role 3.00 (0.00) Proficient inquiry 
Student role 2.00 (0.00) Developing inquiry 
Questioning ecology 2.50 (0.71) Proficient inquiry 
Communication pattern 3.00 (0.00) Proficient inquiry 
Classroom interactions 2.00 (0.00) Developing inquiry 
Global 2.71 (0.40) Proficient inquiry 

SWOT Analysis 


Figure 4 shows the result of the SWOT analysis comparison for the ends of the 2020/2021 and 2021/2022 
academic years. It collects the identification codes for each of the arguments that teachers T1 and T2 gave globally 
to the questions indicated above. 


Figure 4 
Longitudinal Comparison of the SWOT Analysis for the Global Participants 
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Thus, in 2021, within the analysis of internal factors, amongst the weaknesses stated, both teachers coincided in 
highlighting the lack of teaching skills in the conceptualisation of the STE(A)M approach (TCM), withT2, in addition, 
drawing attention to this section to time management (MNG). In contrast, the strengths put forward include teach- 
ing planning capacity (PLN) forT1 and methodology (MTD) forT2. Furthermore, in the analysis of external factors, T1 
contemplates the multidisciplinary nature of the projects (MLT) as a threat, whereas T2 identifies contextualisation 
(CTX) and student skills (STS) in this category. T1 was the only one of the two who specified opportunities, explain- 
ing the development of inquiry skills (INQ), the improvement of student learning results (RES) and the advantages 
associated with the cooperative scenario offered by these types of projects (COO). 

Following this, in the year 2022, it was observed how in the internal factors analysis T1 considered time man- 
agement (MNG) as a weakness, whereas T2, after the projects had been carried out, identified a lack of creativity 
(CRE) for their development. Regarding strengths, it is worth mentioning the fact that T1 describes the cooperative 
scenario (COO), something that in 2021 she considered to be an opportunity, and which it seems finally materialised. 
For her part, T2 continued to consider teaching planning capacity (PLN) as a strength. In the external factors analysis, 
the threats expressed are lack of teaching collaboration (CLB) and student skills (STS), by T1 and T2, respectively. It 
is worth highlighting how in the opportunities expressed in 2022, T1 now mentions the multidisciplinary nature of 
the projects (MLT), something that in 2021 was perceived as a threat, while T2 regards student motivation towards 
learning (MOT) as a positive aspect. 

With the aim of illustrating some of the responses provided by the teachers, Table 5 shows the emergent 
analysis codes, along with a small exemplification of the arguments given. 


Table 5 
Examples of SWOT Analysis Coding 


SWOT Category Code Example 
Cooperative Scenario T1 “I highly value the cooperative and participatory aspect that 
(COO) projects have.” 
Strengths 
Teaching Planning T2 “Be able to choose the line of research and plan one’s own work 
(PLN) so that it is effective in student learning.” 
Time Management T1 “One of the weaknesses found is my time management to develop 
(MNG) projects.” 
Weaknesses 
Creativity sigh ; F : 
(CRE) T2. “It's hard for me to focus and find good ideas. 
Multidisciplinarity Nature (MLT) m1 ‘In my case, it has helped me to have a more global and interdis- 
ciplinary vision of science. 
Opportunities 
Motivation F F . » 
(MOT) T2 “Make the classes more practical and attractive for the students. 
Teaching Collaboration T1 “The main difficulty encountered has been the lack of collaboration 
(CLB) from my teaching team in the classroom.” 
Threats 


Student skills 
(STS) 


T2 “In some cases, the students do not see the importance or are 
not sufficiently prepared/motivated, and this can spread to the rest 
of the group.” 


Teaching Perception of Indagasteam Escuela Project and Collaborative Framework 


The results of the quantitative analysis of Questionnaire 1 for teachers T1 and T2, referring to the design 
and implementation of the projects, are shown in Figures 5 and 6, respectively. This analysis showed that both 
teachers considered that the driving strategies of the intervention employed in the design and implementation 
of the projects developed signified an important methodological change in their science classes ($5), at the same 
time as permitting the stimulation of the efforts of their students and promoting their learning and that of their 
classmates ($1). Likewise, both considered that the implementation of their projects encouraged them to reflect 
on their teaching practice (S7), although T2 did not consider that it improved the learning of her students ($3) or 
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how they related to each other in class (4), in contrast to T1. 

Both considered that the materials provided as support for the design of the projects were of great help for 
the design phase, especially for establishing connections between curriculum and evaluation planning (S8). T2, 
however, did not find those materials of her own design useful for creating learning situations with her students 
(S6), unlike T1. 

Lastly, both afforded great value to the contents worked on in the projects, given they were interesting for 
applying the curriculum to Primary Education in matters of interest for their students (S2). 


Figure 5 
Results of Teacher Perceptions of Projects 
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Regarding the valuation of the established collaborative methodology, both teachers stated that the com- 
munication spaces promoted ($14), relationships established (S10), and resources and strategies provided ($11) 
enabled the transfer of the driving strategies of the project of collaboration and improved throughout their devel- 
opment, being of practical use for their classes. In this regard, in the comparison between the two, there is a higher 
score for T2 than for T1 for the indicated items. Nevertheless, it should be pointed out that T2 did not consider 
improvements in the adjustment of the methodology of the project to her training level (S13), in contrast to T1. 


Figure 6 
Teacher Perceptions of Collaborative Framework 
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Discussion 


The educators in the study, Primary Education teachers, faced a number of different challenges in order to 
conceptualise, design and apply their vision of the STE(A)M approach through the use of inquiry and project-based 
work. To do so, they were provided with a particular strategy of active progressive training over the duration of 
the project. The teacher involvement fosters their self-efficacy and their beliefs, sustained in different degrees of 
scaffolding in STE(A)M teaching domains (Margot et al., 2019; Roehrig et al., 2021; McLure et al., 2021) associated 
with tensions as regards tackling curricular integration, and teaching methodologies orientated towards problem- 
solving and situated learning of their students. The data obtained show different achievements in some personal 
domains, such as the level of competent inquiry reached when implementing the STE(A)M projects they designed 
and aconsiderable improvement in the methodological change incorporated, increasing the degree of autonomy 
provided to their students. 

In response to the first research question raised, anumber of satisfactory results have emerged, given the short 
implementation time of the IndagaSTEAM Escuela project. This is deduced from the evolution observed by the 
researchers/co-authors of this study as regards the teaching inquiry profile of T1 and T2, which evolved positively 
from developing inquiry to competent inquiry (Marshall et al., 2009), facilitator of autonomous and active learning 
in students. Yet, there is still a perception of a certain failure to take advantage of unplanned learning moments, 
with a lack of exploration of ideas and questions arising throughout the projects, something also confirmed with 
the perception obtained in statement $3, which fails to reach positive levels for T2. This could be identified as 
one of the teaching tensions described in the literature, related in this case to methodological aspects (Margot 
& Kettler, 2019). However, it could be seen how there is an improvement in the degree of autonomy provided to 
students ($1) and in the methodological change incorporated ($5), with positive teaching perceptions for both 
educators. Finally, some pleasant results confirm how they positively value the relationships established with the 
different actors (S4), and the processes of reflection promoted (S7), all of which are aspects closely related to the 
incorporation into the inquiry-based teaching profile, a driving approach for the projects. 

Relative to the second research question about their perception regarding advantages and difficulties en- 
countered in the projects, there is an equivalent behaviour regarding a series of aspects at the implementation 
level and the established collaborative framework. In contrast, in others, the teachers show different progression. 

Concerning the advantages, throughout the experience, the teachers recognized them in aspects related to 
teaching planning capacity (PLN) and cooperative scenarios offered by these types of projects (COO). In contrast, fac- 
tors associated with methodology (MTD), development of inquiry skills (INQ), multidisciplinary nature of the projects 
(MLT) and student motivation towards learning (MOT) show different considerations. 

Thus, in 2021, within the internal factors analysis, the strengths put forward include teaching planning capacity 
(PLN) for T1 and methodology (MTD) for T2. On the other hand, T1 specified opportunities, explaining the devel- 
opment of inquiry skills (INQ) and the advantages associated with the cooperative scenario offered by these types 
of projects (COO). Following the year 2022, the internal factors analysis T1 considered strengths, the cooperative 
scenario (COO), something that in 2021 she considered to be an opportunity, and which it seems finally material- 
ized. T2 continued to consider teaching planning capacity (PLN). In the external factors analysis, the opportunities 
expressed in 2022, T1 now mentions the multidisciplinary nature of the projects (MLT), something that in 2021 was 
perceived as a threat, while T2 regards student motivation towards learning (MOT) as a positive aspect. Consider- 
ations that base their assessments on the conductive strategies of the intervention project used in the design and 
implementation of the projects developed, as promoters of an important methodological change in their science 
classes ($5), at the same time that they allowed to stimulate the effort of their students and promote their learning 
and that of their classmates ($1). 

Likewise, the assessment of the collaboration methodology established in the project was considered by 
both teachers, useful for their classes, recognizing the interest in the communication spaces promoted ($14), the 
relationships established (S10), as well as the resources and Contributed strategies (S11) facilitated transferring 
the project's driving strategies. 

Regarding the difficulties in the entire experience, they manifested themselves associated with time man- 
agement (MNG) and student skills (STS). However, the conceptualization of the STE(A)M approach (TCM), the mul- 
tidisciplinary nature of the projects (MLT) and contextualization (CTX), which are shown in the first year, give way 
to factors associated with creativity (CRE) and teaching collaboration (CLB) in the second. As a specific behaviour, 
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in the analysis of external factors, T1 contemplated the multidisciplinary nature of the projects (MLT) as a threat, 
whereas T2 identified contextualization (CTX) and student skills (STS) in this category. 

Following this, in the year 2022, it was observed that in the internal factors analysis, T1 considered time manage- 
ment (MNG) as a weakness, whereas T2 after the projects had been carried out, identified a lack of creativity (CRE) 
for their development. In the external factors analysis, the threats expressed were lack of teaching collaboration 
(CLB) and student skills (STS) byT1 and T2, respectively. It is worth highlighting how, in the opportunities expressed 
in 2022, T1 mentioned the multidisciplinary nature of the projects (MLT), something that in 2021 was perceived as 
a threat, while T2 regarded student motivation towards learning (MOT) as a positive aspect. Regarding the project 
strategies, T2 did not consider improvements in adjusting its methodology to their educational level (S13), unlikeT1. 

This evolution of teacher perceptions reflects behaviours collected in other experiences described in the 
literature, evidencing that if teachers verify in the classroom that the methodological change positively affects 
students, their beliefs are modified, and their perception of self-efficacy increases, enabling them to continue 
introducing changes in their teaching practice (Ariza et al., 2021; Roehrig et al., 2021). 


Conclusions and Implications 


This study has presented a training experience for the STE(A)M education scenario, analysed as a case study, 
from a model of school cooperation sustained with the support of university researchers, developed over the 
course of three academic years in a public early childhood and primary education centre. Concretely, the study 
shows how the STE(A)M approach promoted has been used to inform school practices and teachers’ professional 
development efforts. Contested aspects in STE(A)M education, such as which level of integration of the inquiry 
approach is desirable, what the role of disciplinary content knowledge is in STE(A)M, or how much focus to accord 
to the development of skills, among others, emerged in these different accounts. The impact of the program on 
teachers’ beliefs and practices, as well as its effect on teachers’ self-efficacy to implement the STE(A)M scenario, 
are also discussed. 

The longitudinal study carried out employed a diverse selection of instruments for data collection throughout 
the process (rubrics evaluation, ad hoc interviews and Likert-type scale questionnaires). The methodology for their 
treatment and subsequent analysis has permitted us to discover the levels reached by the teachers as regards the 
use of inquiry and its evolution, along with their perceptions in respect of the achievements and difficulties that 
the collaborative scenario has offered to their students and their performance. 

From a global vision, the STE(A)M education panorama implies substantial changes concerning the teach- 
ing practice, curriculum, school organization, professional development models, and teaching materials, among 
others. All the initiatives are essential in conceptualizing and clarifying educational purposes and methodologies. 

Whatever perspective was selected (STS, PBL or inquiry) is firmly based on what it is already known from 
Science, Engineering, or Mathematics education, with the addition of the potentiality and limits of disciplinary 
integration demanding intensive professional efforts closely linked to design, a high-order professional competence 
for teachers. A new focus with well-designed examples of STE(A)M curricular activities, adapting, and sequencing 
existing teaching and learning units could support progressive STE(A)M teachers’ professional development. Oth- 
erwise, the systemic nature of STE(A)M education involves different levels of collaboration between educational 
administrations, educational research institutions, and teachers and schools. 

Regarding STE(A)M teaching professional development, initiatives are needed to provide useful training 
approaches, with training progressions and adequate scaffolding so that teachers can conceptualize and clarify 
educational purposes to design relevant projects and implement them, addressing both personal and systemic 
challenges of educational institutions. As such, in Spain, there is a necessity of examples of research-based cur- 
ricular proposals that can support teachers in the double effort of adopting the STE(A)M educational framework 
and implementing it adequately. 

For this reason, selecting, adapting and sequencing existing teaching and learning units is highly necessary to 
apply and experience them successfully. This is even more the case if linking it to the “teacher as designer” model. 
For all of the above, still respecting due reservations, being a case study, this experience provides an interesting 
proposal for the professional teaching development of Primary Education teachers in the scenario of STE(A)M 
education for the Spanish context, being extrapolated to the present international scenario of STE(A)M education. 
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Abstract. At the time of the current global 
energy crisis energy awareness education 
in school is increasingly important. 
Textbooks directly helping and influencing 
learning play an important role in this, 
especially in the case of natural sciences. 
Thus, the aim of this research was to show 
how science textbooks help to educate 
energy awareness (EA). Regarding this, 

67 Hungarian science textbooks were 
analysed using a quantitative textbook 
analysis from grade 1 to 12. When studying 
the knowledge on EA, the occurrence of 
the key concepts related to EA included 

in the National Core Curriculum (2020) 
was examined according to content, 
pedagogical and structural dimensions. 
According to the results, the most 
common key concepts were renewable 
energy, and energy saving, while the 

least common was the concept of energy 
crisis. The textbooks studied increasingly 
communicate key concepts in the form of 
definitions in the main textbook text at the 
level of information, moving towards the 
upper grades, and paying less attention to 
the attitude-forming elements. There are 
few tasks that require the application of 
knowledge about EA, especially those that 
also develop digital competence. In this 
form, the textbooks studied are less suited 
to developing an attitude toward EA, which 
does not help in making the right decisions 
in relation to EA. 
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Introduction 


Today's global energy crisis increasingly requires the energy aware- 
ness (EA) of the members of society. To do this, it is not enough to know the 
concepts and knowledge associated with EA, it is necessary to develop an 
attitude in society that will enable us to act decisively in everyday life and 
make responsible decisions in the field of energy use. This process of educa- 
tion must begin in early childhood, and it must be achieved that knowledge 
expressed in words leads to actual behaviour, not as proved by several studies 
(Coker et al., 2010; Fehér & Revakné, 2021; Kénya, 2018a; Revak, 2019; Tayci & 
Usual, 2012; Tortop, 2011), that we say one thing and do the other. EA educa- 
tion is a joint task of education and society, through which the knowledge, 
behaviour and emotional attitude must form a complete whole in the rising 
generation. In addition to the knowledge gained at school, the behavioural 
patterns and examples that our students see outside the walls of the school, 
at home, in the family, and in their immediate and more distant environment 
are decisive factors. In school, the basic organizational form is the teaching 
class, where elements of environmental education, including EA education, 
appear either directly or indirectly, in most cases integrated into science 
subjects. Its content is determined in Hungary in the National Curriculum, 
which appears partly in textbooks and partly in the environmental education 
programme of schools as a general educational goal and task. These environ- 
mental education programmes determine what activities, programmes and 
educational tasks are implemented in the school during classes and outside 
the classroom. Thus, a student's attitude and knowledge in the field of EA is 
the result of a system of multiple effects that interact closely with each other 
(Larson, 2010; Weaver 2002), of which this study shows the content and peda- 
gogical elements of textbooks, including science textbooks, which play an 
important role in the learning process, contribute to EA education. Although 
the study covered Hungarian textbooks the applied textbook analysis method 
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has an international significance since the present study focuses not only on the content related to EA but also on 
the pedagogical and structural elements important in EA education. These elements together facilitate a higher 
level of EA based on the efficient application of textbooks in learning. In order to understand where the textbook 
analysis in the present study occupies a place in the system of textbook research, it is necessary to cover some 
of the research methods and results of the concept and evaluation of textbooks, and also those of the textbook 
analysis of environmental education, including EA education. 


Textbook Concepts 


The definition of textbooks is usually based on their functional role. According to Pingel (2010), textbooks 
convey age-appropriate disciplinary knowledge to students in an understandable way to help them learn. G6mez 
etal. (2014) have also highlighted the learning function of textbooks, while others have emphasized that textbooks 
represent the curricula that communicate knowledge, and develop abilities and skills (Kim & Pang, 2022; Rezat et 
al., 2021). In addition to the requirements set by the curricula, cultural and ideological content must also appear 
in textbooks, which together with disciplinary knowledge and elements that help learning and the development 
of abilities, ensure that students are closer to reality and everyday life (Rotger, 1997). However, all this can only be 
realised with the help and the teaching methods of teachers, who can interpret and encode these textbook texts 
with their expertise and pedagogical skills (Caravita et al, 2008; Zahorik, 1991). Based on the different textbook 
concepts, textbooks are educational aids that, based on the curriculum requirements, help the learning process and 
the personality development of the students with the help of various elements of content, form, and pedagogy. 


Textbook Evaluation Models and Methods 


The first demands for textbook analysis appeared in the 19th century. Until the end of the 1970s, textbook 
research was characterised by textbook revision, textbook reconciliation, and textbook correction, devoid of a 
scientific standard. Textbook research from the seventies to the present day has had a multilateral approach, and 
in addition to the content elements, the format, didactic, pedagogical, social, and scientific characteristics of text- 
books have also been examined (Dardai, 1999). However, even today, these analyses lack the impact assessments 
of textbooks on learning, abilities, and personality development. 

Some of the multilateral approaches have sought totality. In historical order, such a model was the “Bielefeld 
raster’, which was described in 1986 by three professors of the University of Bielefeld, who already clearly considered 
textbook analysis to be a scientific activity. The five dimensions studied by them (metamemory, textbook design, 
branch of science, specialised didactics, education science) and the 24 categories within them were referred to as 
one large examination raster. This model was groundbreaking in its own time, as it also investigated the function 
of metamemory, which plays an important role in the learning process. It was a pedagogically very well-recognized 
textbook analysis model, which was, however, not often used in practice (Dardai, 1999; Kojanitz, 2007). 

The textbook analysis model of Weinbrenner (1995) has also aimed for totality and has considered that text- 
books can be examined in five dimensions by longitudinal (historical) and cross-sectional (comparative) analyses 
using acontent analysis procedure. The dimensions include scientific theory, specialist science, scientific didactics, 
educational science, and design (Dardai, 1999). Weinbrenner’s model has a great similarity to the “Bielefeld raster’, 
with the difference that it does not examine aspects of metamemory but seeks scientific theory analysis instead. 
Thus, the model focuses less on learning as a process. 

Compared to the multilateral models mentioned here, the textbook evaluation model, based on the peda- 
gogical trend that interprets learning as a constructive student activity, represented progress. This gave textbook 
analyses a global, systemic framework, which analysed the content and pedagogical details; the main aspect is to 
help students think and make learning more effective. The primary representative of the constructive approach 
is the work of Chambliss and Calfee (1998), which has formulated three main approaches to textbook production 
and evaluation: comprehensibility, exemplary curriculum, and student-centred instruction. According to their 
proposal, the examination of the content, pedagogy, and structural sections of textbook analysis should also be 
subordinated to these three aspects (Dardai, 1999). This model is still used in textbook analysis research to this day, 
providing a framework for the evaluation of textbooks. According to this framework, the primary consideration 
in the content, pedagogical and structural analysis is whether it helps or hinders the learning process, knowledge 
acquisition and personality development of students. 
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Recent textbook research today, following the textbook analysis model based on constructive student activ- 
ity, mainly consists of the analysis of concepts in the material, as well as how these concepts are integrated into 
the textual and non-textual elements, what is the visual presentation of the content to be learned, and how the 
didactic processing of the curriculum helps learning (Dardai, 1999). 

Vojir and Rusek (2019) in their systematic review, have grouped publications on science textbook analysis 
published between 2018 and 2020, based on evaluation criteria. In the list they have collected, the subjects of 
studies with the learning process in mind have been the following: “relationship with curricula, learning, social 
relations of the natural sciences, the historical development of textbooks, e-textbooks and open textbooks, errors 
and misconceptions in textbooks, comparison of textbooks from different geographical areas, textbooks from 
teachers’and students’ point of view (Yilmaz et al., 2017). Among the studies, an increasing number of analyses can 
be found focusing on the forms, in which sustainability and climate change occur in textbooks (Roman & Busch, 
2016; Simsek, 2011). 

The concepts and methods of content and structure analysis can often be encountered in the textbook litera- 
ture. According to Weber (1990), the content analysis procedure as a research method uses a series of procedures 
to draw conclusions from the text. Examples of such methods are coding schemes and their testing, computerized 
text classification, the creation of reliability and classification schemes, assumed and inferred categories, alterna- 
tive classification systems, and single or multiple classifications. Weber’s method of content analysis suggests both 
qualitative and quantitative methods of analysis, as does a much earlier concept of content analysis that originated 
from Berelson (1952). According to it, content analysis means structuring, categorisation, and qualitative and quan- 
titative evaluation according to specific objective criteria. The majority of research that performs content analysis, 
like those mentioned here, categorizes according to specific criteria and looks for the frequency and place of a 
given category in the text (Devetak & Vogrinc, 2013; Qadeer, 2013). 

There are usually three levels of structure analysis: general structure, text analysis, and visual content analysis. 
When evaluating the general structure, they include page numbering for the chapters, the size of the chapters, the 
ratio of text and illustrations, etc. In the course of text analysis, it is studied whether there is a fragment of the text 
that encourages teacher explanation, teacher demonstration, student assignments, experimentation, problem- 
solving, or the book contains a section on repetition and summary in the chapters, a collection of concepts at the 
end of the book. For example, the structural analysis of the visual content of the book discusses the presence and 
proportion of photos, drawings, graphics, tables, the ratio of visual elements and text fragments (Abubaker & Lu, 
2011; Devetak & Vogrinc, 2013). 

It can be seen that both content and structural analysis imply the possibility of qualitative and quantitative 
evaluation, both separately or in mixed form. Qualitative evaluation is an analytical method with much more 
subjective elements than quantitative evaluation, which relies more on the analyst's own value system and un- 
derstanding of the text. In the course of this method, it is essential to accurately define the categories that are the 
subject of the analysis (e.g., what aspects of the history of science are in the textbook, the analysis of the scientific 
correctness of the concepts, what elements of the formation or maintenance of the type of misconception can be 
observed, what types of errors occur) (Pingel, 2010). 

In the course of quantitative analysis, the frequency of the textbook element under consideration and/or the 
magnitude and extent of its position in the text are most often examined (Pingel, 2010). For this analysis, one, two 
or more scientists can perform the coding of the studied elements. Hardly any examples of coding comparison can 
be found in the literature of textbook evaluation, most of the time it is not even mentioned how many people did 
the coding. For example, the publication of Kim and Pang (2022) on the analysis of the content on sustainability 
occurring in primary school mathematics textbooks mentioned that coding was performed by two encoders and 
Interclass Correlation Coefficient (ICC) was used to test code matching. 

The two evaluation methods have been used simultaneously by the vast majority of textbook analysts, as the 
two methods together give a more complete picture of the content, structure, pedagogical characteristics of the 
textbook being studied (Kim & Pang, 2022). 


Research Studying the Occurrence of Environmental Education and Energy Awareness in Textbooks 
Environmental education content appeared in textbooks as early as the end of the 18th century, however, 


their textbook study intensified only in recent decades. For example, under the 1992 PHARE (Poland and Hungary 
Assistance for Restructuring their Economies) programme, 201 books with environmental content have been 
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analysed. The results have revealed that the approach to environmental education hardly appeared in textbooks. 
Children rarely encountered with sustainability. The study has also shown that teachers did not take advantage of 
the specific methods of environmental education, but rather traditional teacher-centred methods (e.g., teacher 
performance, explanation, discussion) were typical. 

Later research has mainly focused on the analysis of the environmental education aspects of science textbooks 
(Kénya, 2018b; Kojanitz, 2003; Oguz et al., 2004; Salmani et al., 2015). One of the largest programmes was BIOHEAD- 
CITIZEN (Biology, Health and Environmental Education for Better Citizenship) 2006. This programme has brought 
together 19 countries to examine areas of environmental and health education and biology education. The aim 
of the research has been to formulate recommendations based on the results for improving teaching on biology, 
health, and environmental topics and for policymakers. The analyses have had a special focus on the examination of 
illustrations in the textbooks and the analysis of the curriculum according to whether it was linearly or non-linearly 
constructed (Horvath et al. 2009). An important conclusion of the programme has been that environmental topics 
were covered in a small number of pages in the books, which would be necessary to increase in order to acquire 
appropriate knowledge and even more so to shape the environmental attitude. The low level of illustrations related 
to environmental topics has also been cited as a problem, which has been a major shortcoming in understanding 
environmental topics and problems, motivating and raising awareness for these topics. In other words, environ- 
mental education through textbooks is more of a knowledge than an attitude-forming nature (Kénya, 2018b). 
Further textbook analyses linked to different countries can be found in the international literature, which studies 
the environmental education content of textbooks and the presence of ecological knowledge in them. Kojanitz 
(2003) for example has examined the extent to which geography books in Hungarian vocational schools met the 
requirements of environmental education. In this textbook analysis, the statistical, data-like recording of the en- 
vironmental aspects of the textbooks has been performed. Kojanitz (2003) counted the lines that were related to 
environmental science obtaining in this way a quantitative indicator that allowed the determination of the ratio of 
environmental aspects to the total textbook content. Salmani et al. (2015) conducted a content analysis of newly 
prepared Iranian social and natural science textbooks. The study has covered the topics of ecology, environmental 
protection, and human activity. The analysis has focused on both content and structural elements, its unit was the 
textbook page itself, with text, images, questions, and exercises. The results have shown that in both social and 
natural science textbooks, the topic of ecology was dealt with to the greatest extent, followed by human activity 
and then environmental protection. 

Most of the research on EA education has studied the knowledge and attitude of students related to renewable 
energy. Coker et al. (2010) measured the knowledge of Turkish primary and secondary school students on renewable 
energy, which showed that students had many misconceptions regarding renewable energy. The content of the 
studied science textbooks related to renewable energy is also responsible for these misconceptions. Knowledge is 
communicated, in most cases, in short definitions that do not describe the context. Tortop also conducted studies 
with Turkish high school students in 2011. The research has shown that students did not have the right knowledge 
and attitude towards renewable energy sources, they had more misinterpretations, and they did not have accurate 
knowledge, for which textbooks could have also been blamed. 

The number of textbook content analyses focusing on multiple aspects related to EA is small in the interna- 
tional literature. In Hungary, Uténé-Kiss, (2012) and Revakné et al. (2018) have studied the content, structural, and 
pedagogical characteristics of energy awareness education in natural science and geography textbooks in primary 
and secondary schools. The main conclusion of their study has been that in the analysed textbooks, EA education 
was much more knowledge-like, with little content and structure related to attitude shaping. Similar results have 
been obtained by Chao et al. (2017), who have studied the knowledge, attitude, and behaviour of high school 
students learning from two different geography textbooks in relation to energy saving and carbon reduction. The 
results have revealed that the proportion of knowledge elements in the geography textbooks was much greater 
than that of attitude and behaviour. The study has applied content analysis by searching key concepts related to 
energy saving and carbon reduction and analysing their ratio related to the text of the whole textbook. 


Research Problem and Focus 
The basic problem studied by the present research, based on previous research results, is that the energy 


awareness of students is lower compared to factual knowledge level, which can be explained by the fact that, apart 
from other influencing factors, textbooks (primarily in natural sciences) do not emphasise EA education enough 
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in content, pedagogy and formal aspects (Coker et al., 2010; Fehér & Revakné, 2021; Konya, 2018a; Revak, 2019; 
Tayci & Usual, 2012; Tortop, 2011). 
Regarding the above problem, content analyses were made previously in relation to the project entitled “The 
Role of social learning processes regarding renewable energy resources in the Eastern Region of Hungary” (sup- 
ported by the NKFIH, K 116595 application in Hungary, 2016-2019). These, however, included only natural history 
(grades 1-6) and geography (grades 7-9) textbooks. The aim of the research is to continue this textbook analysis 
after 2019 and extend it to every textbook in natural sciences distributed in Hungary. 
The questions of the research were the following: 
1a) What is the proportion of knowledge related to EA specified in the National Curriculum (2020) in the 
studied science textbooks? 
1b) Secondly, in analysing the professional content dimension, the description and determination of 
knowledge related to EA and the distribution of knowledge according to the determination methods 
are examined. 
2a) Analysing the pedagogy dimension of the textbooks, the first question is what didactic elements are 
used in deepening and applying knowledge related to EA in the studied textbooks? What are the ratios 
of the key concepts among the didactic elements? 
2b) Analysing the pedagogy dimension of the textbooks the second question is what is the ratio of EA-related 
attitude development in the studied textbooks and what is the ratio of occurrence of the examined 
knowledge in these parts? 
3) Analysing the structural dimension of the textbooks the question is what structural units are typical in 
the studied textbooks and what is the ratio of EA-related knowledge among them? 


Research Methodology 
General Background 


The theoretical framework of this textbook analysis was provided by a textbook evaluation model that was 
based on a pedagogical trend that interprets learning as a constructive student activity. In this way, the textbook 
analysis focuses on how textbooks help the learning process and the shaping of the way of thinking of students 
(Chambliss & Calfee, 1998; Vojir & Rusek, 2019). In the course of the research, the content that meets the EA require- 
ments of the 2012 and 2020 National Curricula, including the appearance and definition of knowledge and concepts 
of EA, was analysed in the spirit of comprehensibility and straightforward curriculum planning. The occurrence of 
knowledge and concepts related to EA was also subjected to structural and pedagogical evaluation, considering 
how much these help students to learn and develop their attitude towards EA. The evaluation was carried out 
using a quantitative content (conceptual) analysis method, evaluating content related to EA from professional, 
pedagogical, and structural aspects. 


Sample 


The present research was carried out between 2018 and 2021, during which currently available science 
textbooks written for grades 1-12 were analysed in Hungary in relation to EA education. The selection was based 
on the officially licensed and most commonly used textbooks in public education included in the List of public 
education textbooks 2018/2019 (Educational Authority, 2018). Thus, 67 science textbooks from five publishers 
distributing them were analysed. Students study natural history in grades 1-6, biology in grades 7-12, chemistry 
in 7-10, geography also in grades 7-10, and physics in grades 7-12, therefore the studied textbooks were analysed 
in the distribution of the given subjects according to these grades (see Table 1). The content of the textbooks is 
based on the requirements of the National Curriculum (2012 & 2020) for the given subject and for environmental 
and EA. Among the textbooks, four groups of textbooks were found (two in biology, one in geography, and one 
in natural history), where the same author wrote textbooks for different grades (even three or four grades) of the 
subject in question. In other cases, one author wrote textbooks for one or two grades within a given subject. 
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Table 1 


The Number of Studied Textbooks (n) According to Grades and Subjects 


Grade 
Subject ——— a 
1 2 3 4 5 6 7 8 9 10 11 12 
Natural history (n) 4 4 3 1 4 4 - - - - - - 20 
Biology (n) - - - - - - 2 2 2 3 2 2 13 
Chemistry (n) - - - - - - 2 2 2 3 9 
Geography (n) - - - - - - 3 5 3 3 14 
Physics (n) - - - - - - 1 2 2 2 2 2 11 
> (n) 4 4 3 1 4 4 8 1 9 1 4 4 67 
Procedures 


In the course of the study, after formulating the goals and questions of the research, the theoretical framework 
of the textbook analysis was determined as a first step. The National Curriculum (2012 & 2020) is fundamentally 
built on the constructive pedagogical direction when designing the requirements. Therefore, the multidisciplinary 
textbook evaluation model that emphasises the learning process as an active activity as the basis for analysis was 
also considered (Chambliss & Calfee, 1998; Vojir & Rusek, 2019). Based on this, professional content, structural and 
pedagogical analysis dimensions were studied, bearing in mind in each case how the studied dimensional ele- 
ment contributed to the learning process (see Figure 1). The method applied in the study was quantitative content 
analysis and conceptual analysis within it. The occurrence frequency of key concepts related to EA was analysed 
along professional, pedagogical and structural dimensions. 


Figure 1 
Outline of the Research 


Textbook analysis methods preferring a 
constructive learning procedure 


Analysis from professional aspect Analysis from pedagogical aspect Analysis from structural 
a) Searching for knowledge related a) Applying knowledge related to EA aspect 
to energy awareness (EA) b) Shaping attitude Knowledge related to EA 
b) Type of description, definition in structural units 


In the course of conceptual content analysis from a professional aspect, first, the findability of knowledge re- 
lated to EA in textbooks was examined. The search was based on the EA requirements of the National Curriculum 
(2012 & 2020), which defined what key concepts related to EA should be discussed during the learning of natural 
sciences within the framework of environmental education (in Hungary, the elements of environmental education 
appear mainly integrated into natural science subjects.). In order to make the scope of knowledge relevant to EA 
specific and well-defined, the following key concepts were searched: renewable energy, energy saving, heating, 
power station, conscious energy use, energy crisis, energy efficiency, and energy consumption. The presence of 
the particular key concept in a given textbook was examined. It was repeatedly observed that it was not the key 
concept itself that appeared in the textbook, but some knowledge about it. For example, it was not the concept 
‘renewable energy’ that appeared, but wind or hydropower in the atmosphere or hydrosphere topics. In this case, 
it was considered that the concept of renewable energy appeared figuratively more than once in the textbook, 
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i.e. in it the occurrence rate of the given key concept was greater than in a textbook where the given key concept 
was found in only one case (e.g. a definition) The occurrence of each key concept in the entire textbook sample 
was studied, and then occurrences were examined by grades and subjects as well. 

Those cases when no knowledge related to EA could be found in the studied textbook were eliminated from 
the data set. Thus, a data set without cases with zero occurrences was used for the assessment. In the course of 
the content analysis, the way in which the key concepts were described and defined was examined again. Two 
aspects seemed reasonable, either definition-like or indirect specification of the concepts. This categorization was 
carried out because, according to previous studies (Adamik et al. 2004; Nagy Varga, 2014) the process of acquiring 
knowledge and understanding processes could be influenced by the fact that the given concept was present in 
textbooks in the form of a short, specific, to the point definition or longer text with explanatory elements, in a 
circumscribed way. How this concept definition influences the factual knowledge of students could be the subject 
of future research. The ratio of key concepts occurring as definitions and circumscriptions in the total sample, as 
well as by grade and subject. In analysing the occurrence of the key concepts according to these two aspects, 
as well as when examining them in additional dimensions, the co-occurrence of all key concepts and not single 
concepts was examined based on specific criteria. (Examining each key concept would be the subject of separate 
studies. Here we would like to give a general picture with a focus on EA). With regard to the key concept occur- 
rences according to the way they are described and defined, the fact whether the concept can be found in one 
or both (both: definition and circumscription as well) definition types in the given textbook was looked at. One 
of the options for acquiring and recording knowledge more deeply in textbooks is the appearance of the given 
knowledge in several content or didactic contexts. 

As didactic aspects and elements of the pedagogical dimension, three variables were analysed: textbook 
tasks, homework tasks and tasks that require internet application. The cognitive, social, etc. abilities these tasks 
develop were not studied. This may be the subject of a different research. The occurrence of key concepts for the 
total number of cases was examined in this case as well, which was the fewest here. This is the result of the fact 
that even if a concept occurred in the textbook, the task for its application may not occur in the textbook. Thus, this 
further reduced the number of cases that can be investigated. Analyses by both grade and subject were carried 
out, as well as co-occurrence studies (whether the key concept occurred in one, two or all three types of tasks). 

In the course of the pedagogy analysis, the analysis of the distribution of the key concepts was the next 
according to the aspects of attitude shaping, with the aim of seeing the extent to which the studied textbooks 
contributed to the development of the attitude related to EA. Two aspects seemed logical for the analysis. One was 
the textbook elements of attitude shaping itself (textbook sections on behaviour, emotion, belief, and attitude) 
and the other was the occurrence of the given key concept in elements that relate only to its factual knowledge 
acquisition. The total number of occurrences of the key concepts actually found in the textbooks, as well as their 
occurrence by grade, subject and co-occurrence, according to the textbook sections that provide attitude-shaping 
and factual knowledge were also studied. 

In the structural analysis dimension, the structural units were examined in which the given key concept or 
knowledge related to it occurred (Devetak & Vogrinc, 2013). The structural units studied were the following: 1) 
Main text (mandatory concepts, knowledge defined by the curriculum, which composes the main unit of teaching 
and learning in textbooks); 2) Reading (additional to the main text and can be clearly distinguished from it, which 
serves to supplement and motivate the main text); 3) Complementary text (non-compulsory units containing minor 
additional information formally separated as part of the main text); 4) Figure, image (its purpose is to illustrate the 
content of the main text, to help understanding, to motivate). This categorization of the structure was justified 
by the fact that the learning process can be influenced by the occurrence of concepts and knowledge in a given 
type of textbook structure element. What students are sure to read and learn is the main text of the curriculum. If a 
given concept or knowledge is present only in reading, in additional text or only in the form of a figure or picture, 
the students do not pay special attention to it, they will overlook it if the teacher does not specifically draw at- 
tention to it. Presumably, the most effective way of learning is when students encounter a given knowledge in as 
many structural elements as possible at the same time. Based on these aspects, all the occurrences were studied 
by grades and subjects and the co-occurrence of the key concepts in the given structural units was also analysed. 

All occurrences were given 1 point but if the key concept was missing 0 point was given. As a result, dichoto- 
mous variables were used in all cases. 

Analyses were carried out by two scientists (who studied all of the textbooks, involved in the research) and 10 
teachers (two teachers for each subject who analysed only the textbooks of their own subject). The results of the 
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two scientists and the two teachers were compared by subjects. As a result, the similarity of the data (considered 
categorized items) was studied by several coders. Considering correspondence, the Fleiss’ kappa value was .94 - .97 
(Natural history: .94; Biology: .95; Physics: .97; Geography: .96; Chemistry: .97). Key concept occurrences where two 
or more differences were found between the data of the four analysers were deleted. 


Data Analysis 


For the occurrence of key concepts (with the exception of co-occurrence according to the given criteria, where 
the Valid Percent values of the frequency test were used), relative frequency values were given. The occurrence 
of key concepts was given 1 or 0 points on a dichotomous scale (based on whether it occurred according to the 
aspect under consideration). 

Cochrans’Q test was performed: 1) in the evaluation in each dimension, to analyze the distribution by key 
concept according to the given aspect for the total sample (when searching for key concepts, all cases studied 
in all textbooks, including if there was no occurrence in the given textbook); 2) in the content, pedagogical and 
structural dimensions, to analyze the distribution of key concepts ( co-occurrences of key concepts) according to 
more than two aspects within a grade or subject. 

The Chi-Square test of the Crosstabs option in Descriptive Statistics was used: 1) to examine the distribution 
of each key concept (each key concept separately) by grades and subjects; 2) to analyze the occurrence of key 
concepts (key concepts together) in specific aspects of a given dimension by grades and subjects. 

Wilcoxon test was used to evaluate: 1) the content, pedagogical, and structural dimensions according to two 
criteria for the total number of cases that take into account actual occurrences; 2) key concept distributions within 
dimensions according to two aspects within grades and subjects. 

The Frequencies option of Descriptive statistics was used to examine the percentage of key concept (Valid 
Percent) occurrences, depending on the dimension, once (in one aspect, e.g., as a definition or circumscription 
within a conceptual dimension) twice (e.g., as both definition and circumscription) or in multiple aspects (triple or 
quadruple occurrence). When analysing these single or multiple occurrences, only differences within grades and 
subjects were examined, for which the Chi-Square test was used. Statistical analysis of the data was carried out 
using SPSS version 26.0. Effect size (d) measurements were made for each statistical test. 


Research Results 
Conceptual Content Analysis from a Professional Aspect 


The first question of the research concerned conceptual content analysis from a professional aspect. For 
the answer, the occurrence of key concepts related to EA specified by the National Curriculum (2012 & 2020) was 
examined in the studied textbooks. 

First, the relative frequency (see Table 1) of each key concept was determined for all studied textbooks and 
occurrence numbers (N = 621) (total sample) (see Table 2). 


Table 2 
Relative Frequency and Standard Deviation of Key Concepts Related to Energy Awareness in the Total Sample and for the 
Number of Occurrences (N = 621). 


Renewable Energy Heatin Power Conscious Energy Energy Energy 
energy saving g station energy use consumption efficiency crisis 
M 0.75 0.38 0.52 0.66 0.40 0.42 0.12 0.63 
SD 0.43 0.49 0.50 0.47 0.49 0.49 0.33 0.48 


The total number of cases tested for all occurrences in the total textbook sample was N = 621. The difference 
between the relative frequency values of key concepts was significant (Cochrans’Q = 65.48; df = 7; p < .001;d= 
0.686). Based on the d value, the distribution in textbooks showed an intermediate effect. The concepts of renew- 
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able energy, energy crisis and power station appeared in the largest proportions, followed by heating, energy 
consumption, conscious energy use, then energy saving and energy efficiency. 
The distribution of the above key concepts by grades was also studied (see Table 3). 


Table 3 


Relative Frequency of Key Concepts Related to Energy Awareness According to Grades (N = 621) 


/Print/ 


can Pi Ea gy Fa Sms oy oa gp 
1; 33 33 .00 .00 .00 33 .00 .00 5.71 7 573 1.117 large 
2. .00 33 .00 .00 33 .00 .00 .00 8.11 7 322 1.188 large 
3. 33 33 33 33 33 33 33 .00 2.66 7 914 0,706 intermediate 
4. .00 1.00 .00 33 1.00 .00 33 .00 8.00 7 333 0.752 _ intermediate 
5. 86 50 25 Gi .00 .00 .00 00 19.03 7 .008 2.180 large 
6. 1.00 1.00 1.00 1.00 1.00 .00 .00 00 22.00 7 .003 2.625 large 
7. 95 50 63 84 55 66 13 56 25.40 7 001 1.151 large 
8. 81 10 63 89 40 63 M1 44 25.86 7 .001 1.114 large 
9. .90 25 15 81 33 22 .00 44 3.,30 7 <.001 2.106 large 
10. 13 .08 54 42 40 63 28 55 2.13 7 <.001 1.021 large 
11. 60 25 .00 50 .00 .00 .00 50 1.06 7 185 1.353 large 
12. 1.00 50 AQ .66 1.00 50 50 50 5.91 7 541 1.474 large 
oa 32.18 13.68 21.05 39.25 16.17 21.20 9.69 13.29 
df 11 11 11 11 11 11 11 11 
p < 001 251 033 < .001 135 031 558 215 

Effect 1.237 1.108 1.353 1.359 1.148 1.359 0.885 1.003 

size(d) large —_ large large large large large large —large 


Taking the key concepts together, there was a significant difference in the distribution by grade (7 = 86.42; 
df= 11; p < .001; d= 0.804). Based on the d value, the impact of grades was large in terms of the relative frequency 
of the key concepts. Analysing the distributions in detail (see Table 3), there were significant differences in the case 
of renewable energy, heating, power station and energy consumption according to grade. The concept of renew- 
able energy and the knowledge associated with it were most prevalent in grades twelve, six, seven, nine, and then 
five and eight. The lowest relative frequency values were observed in grades 1-4. According to the occurrence of 
key concepts, two large groups could be distinguished. One was grades 1-4, where zero or low relative frequency 
values were typical, and the other was grades 5-12, where these values were much higher. 

In the case of the key concept’heating; similar two groups could be distinguished. Grades 1-4 with low or zero 
relative frequency values and grades 5-12 (except for grade 11, where there is zero occurrence) with higher values. 
The greatest occurrence of the latter was observed in grade six, and then in grades nine, seven, eight, ten and twelve. 

In the case of the key concept ‘power station, the same two groups with similar relative frequency values 
were observed as in the case of the previous two concepts. This concept was present in all the textbooks studied 
in grade six. Then grades eight, seven, and nine followed, between which there was no significant difference 
(7 = 1.55; df= 2; p = .460), and grades twelve, eleven, and ten brought up the rear. In the case of ‘energy consump- 
tion, the presence of this key concept was negligible in grades 1-6, and in grades nine and eleven, while it was 
present with a greater proportion in the other grades. 

Analyzing distributions within each grade, students in grades 1-2 encountered knowledge related to ‘renew- 
able energy’ ‘energy saving; ‘conscious energy use’ and ‘energy consumption’ while in grade three all key concepts 
appeared except for’energy crisis: In grade four, the concepts ‘renewable energy’and ‘heating’ were missing. Grades 
five and six were significantly (p < .05) lacking the knowledge of ‘energy crisis; ‘energy consumption, and ‘energy 
efficiency’, while in other grades, except for grade eleven, all key concepts occurred. In these latter grades, the 
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knowledge of ‘renewable energy; ‘power station, and ‘energy consumption’ occurred in the highest proportion, 
while ‘energy efficiency’ and ‘energy crisis’ have the smallest proportion. 

Overall, the concept ‘energy saving’ was encountered in all grades, followed by the concepts ‘renewable 
energy’ and ‘power station, while the lowest occurrence was shown by ‘energy efficiency’ and ‘energy crisis, which 
was especially typical in grades 7-12, however, with values below .56 relative frequency there as well. 

The occurrence of key concepts related to EA was examined according to subjects as well (see Table 4). 


Table 4 


Relative Frequency of Key Concepts Related to Energy Awareness According to Subjects (N = 621) 


Subject Renewable Energy Heating Power Conscious Energy Energy Energy Fe df p d Effect size 
energy saving station  energyuse consumption efficiency crisis (a) 

Newel CS 50 33 06 12 00 2335 7 001 0924 — large 
history 
Biology 42 23 38 22 30 ee 08 29 546 7 604 0473 — small 

. < 
Physics 84 545 88 30 50 00 7 351 7 gy yang large 
Geogr 400 1.002 94 66 15 33 66 650 7) (0.328 large 
phy 001 
nn eC 70 66 22 50 A 2 945 7 419 0.721 intermedi 
istry ate 
3740 218 1061 40.54 687 18.18 5.85 23.63 

af 4 4 4 4 4 4 4 4 

p <.001 702 031 <.001 143 001 210 © <.001 

d 1.387 0.397 0.866 1.390 0.685 1219 0629 1.493 

ra large small __large lage Meret stg interme isa 
size (d) g g g ate 9 diate g 


Taking the key concepts together, there was also a significant difference in the distribution by subject 
(7 = 114.92; df= 4; p < .001; d = 0.953). Value d again indicated here that the occurrence of key concepts was also 
strongly influenced by subjects. When analysing each key concept by subject, significant differences were detected 
in the case of the concepts ‘renewable energy; ‘heating; ‘power station; ‘energy consumption’ and ‘energy crisis’ 
according to each subject (see Table 4). Apart from the concepts ‘heating’ and ‘energy crisis, the highest relative 
frequency values of the key concepts could be observed in the textbooks of the geography subject, followed by 
physics, then by the values of chemistry and biology textbooks. In the case of key concept distributions indicating 
significant differences, the influence of subjects showed a large effect (see Table 3). 

Considering the relative frequency values of key concepts within given subjects, significant differences 
could be found in the textbooks of the subjects of Natural history, Physics, and Geography (see Table 4). In the 
textbooks of natural history, the concepts ‘renewable energy’, ‘power station’ and ‘energy saving’ were encoun- 
tered the most, while the concepts ‘energy efficiency’ ‘energy consumption’ and ‘energy crisis’ occurred in the 
smallest number. In physics, ‘conscious energy use’ had a lower relative frequency, while ‘energy efficiency’ was 
completely absent from physics textbooks. The other key concepts showed a relative frequency value of 0.50 
or more. Textbooks of geography showed a dominant role in the occurrence of key concepts. For example, 
knowledge of ‘renewable energy’ and ‘energy saving’ was present in all studied geography textbooks, but the 
concept ‘power station’ also had a high relative frequency (M = 0.94). ‘Energy efficiency’ was discussed in the 
lowest proportion in geography textbooks (M = 0.33), but the relative frequency of all other concepts was above 
0.60. In the textbooks of the subject of chemistry and biology, all concepts could be found. In biology, each 
occurrence was below 0.50. In chemistry, ‘renewable energy’, ‘heating, and ‘power station’ showed occurrence 
values higher than 0.50. With the exception of biology and chemistry, the occurrence of key concepts within 
subjects also showed a large influence which indicated a significant influence of subjects on the occurrence 
of key concepts. 
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Overall, it can be observed (see Table 4) that all the key concepts occurred in all subjects except for ‘energy 
crisis’ and ‘energy efficiency’ The concept and knowledge of ‘energy crisis’ were absent from the textbooks of the 
natural history subject, while ‘energy efficiency’ was missing from physics textbooks. Geography textbooks domi- 
nated in terms of occurrences. 

In the second part of the first research question (1b), it was analysed how key concepts relating to energy 
awareness were defined in a given textbook, as a specific definition, circumscription, or both. In this analysis, only 
those cases were examined where there was an actual occurrence, while textbook cases where the occurrence of 
key concepts was not observed were eliminated. Thus, out of a total of 621 cases, 320 cases could be investigated 
further. The analysis here was no longer carried out by key concepts, the co-occurrence of all concepts was inves- 
tigated according to the method of definition. 

For the 320 cases, the relative frequency of definition (M = 0.68; SD = 0.46) and circumscribed (M = 0.34; SD = 
0.47) type descriptions showed a significant difference (Z = - 6.174; p < .001; d= 0.733) with intermediate impact 
size. The obtained result clearly shows that the key concepts were described more in the form of a specific defini- 
tion regarding all occurrences. 

The distribution of key concepts by grades and subjects was examined here as well. Studying the definition 
description, the effect of grades (y?= 31.60; df=11; p = .001; d= 0.636) on the textbook occurrence of key concepts 
overall was intermediate and significant. 

In the distribution of definition type descriptions by grade, it was found that, except for grade four (M = 0.33), 
the relative frequency values were above 0.50 (see Table 5). Outstanding is grade eleven, where this value was 1.00, 
i.e., all key concepts were present in the form of definitions. 


Table 5 
Relative Frequency of Key Concepts Related to Energy Awareness According to the Method of Identification by Grades (N =320) 


Grade Definition Circumscription Z p Effect size (d) 

1. 40 60 -.45 655 0.411 small 
2. 50 87 “1.34 .180 1,076 large 
3. 50 83 -1.00 317 0.894 large 
4. 33 1.00 -1.00 317 0.894 large 
5. 17 23 -1.94 .052 3.489 large 
6. 87 13 -2.84 005 2.864 large 
7. 62 43 “1.72 085 0.420 small 
8. 74 27 -3.97 < .001 1.078 large 
9. 84 16 474 < .001 1.819 large 
10. 54 AT -.59 558 0.137 small 
11. 1.00 .00 -2.23 025 0.547 intermediate 
12. 60 40 -0.63 527 0.407 small 
Xx? 31.60 31.66 
df 11 11 
p 001 001 

Effect size (c 0.636 0.663 

intermediate intermediate 


For the circumscriptions, the distribution by grade for 320 cases also showed a significant difference (y? = 
31.66; df=11; p =.001; d= 0.663) and indicated the intermediate effect of the grade on the distribution of key con- 
cepts in this type of description. Compared to the definition description, occurrence rates here were the opposite. 
Only a circumscription was found in grade four, while no circumscription was found at all in grade eleven. Relative 
frequency values were above 0.60 in grades 1-3, and below 0.50. in other grades. 

Considering the two types of description within the grades, circumscription dominated in grades 1-4, while 
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definition dominated in higher grades. This trend showed significant differences in favour of definitions in grades 
six, eight, nine, and twelve. 

The influence of subjects on the textbook occurrence of key concepts (7? = 4.40; df = 4; p = .354; d= 0.211) 
in the case of definition type description was small overall and this effect was not significant. A similar result was 
obtained for the circumscription type description (7? = 5.94; df= 4; p =.203; d= 0.275) (see Table 6). 


Table 6 
Relative Frequency of Key Terms Related to Energy Awareness According to the Type of Description by Subjects (N=320) 


Type of Natural F ; : 2 Effect size 
description history Geography Biology Physics Chemistry x df p (a) 
Definition 13 65 68 18 62 4.40 4 354 0.211 small 
chili 38 36 32 22 45 5.94 4 203 0.275 small 
Z 2.53 3.70 1.89 4.18 96 
p 011 <.001 .059 < 001 336 
0.848 0.600 0.765 1.365 0.365 
Effect size (d) 
large intermediate intermediate large small 


The picture here is quite clear. In the case of all subjects, the definition type description was more frequent 
(see Table 6). There was a significant difference between the relative frequency values of the two types of de- 
scription in the textbooks of natural history, physics, and geography subjects. This shift was best observed in the 
textbooks of natural history and physics subjects. In the case of the rest of the subjects, this value was around M = 
0.60. Analyzing the relative frequencies within the definition and circumscription type descriptions, there was no 
significant difference in the distribution of key concepts by subject in either case. Overall, definition-type descrip- 
tions dominated the above circumscriptions in terms of both grade and subject. 

Lastly in the content analysis, it was examined whether the key concepts occur with one or both types of de- 
scriptions simultaneously in the given textbook. For all cases of occurrence (N = 320), 307 cases were found where 
the occurrence of the concept was single (either definition or circumscription) and 13 cases where it was double 
(in both types of description) (y? = 273.80; df= 1; p < .001; d= 4.868, large effect). 

Studying co-occurrence by grade (y? = 100.89; df= 11; p <.001; d= 1.296), a strong influence of grades on single 
or double occurrences of key concepts was found. Only one type of description was typical in the occurrence of 
key concepts in grades five, six, nine, eleven, and two, while in the other grades, both single and double. There was 
a strong significant difference in grades seven (p < .001) and eight (p < .001) in favour of the single occurrence. In 
contrast, in grades 1-4 there was no significant difference, single and double occurrences had a similar proportion. 

Co-occurrence by subject (y? = 17.77; df= 4; p < .001; d=0.484) indicated a small effect size for subjects. In biol- 
ogy, physics, and chemistry there were only single occurrences, while in geography and natural history, both single 
and double occurrences were found, but the single occurrence was more typical in the case of the latter subject. 


Content Analysis from Pedagogical Aspect 


Question 2a of the examination of the pedagogical dimension concerned what didactic elements are used 
to deepen and apply knowledge related to EA in the textbooks studied, and what proportions can be observed 
based on this aspect in the distribution of key concepts between the individual didactic elements. 

Three typical didactic elements were found in textbooks: 1) textbook tasks that can be solved in class, 2) inde- 
pendent homework, and 3) tasks that require an internet application. Thus, during the evaluation, the key concept 
distribution according to these types of tasks was studied. The number of cases here was 115. This was obtained by 
subtracting from the number of actual occurrences in the textbooks (320 cases) the number of cases where there 
was no task for the given key concepts (205 cases). Similarly to content analysis, a distribution test by task type 
for the total number of cases (N= 115) was carried out and the distributions by grade and subject were analysed. 

In examining textbooks by grade, with the exception of grades seven, eight, nine and ten, the total number 
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of tasks in the other grades, taken together for the three types of tasks, was no more than seven per grade. When 
these small numbers of occurrences were studied in proportion to the total number of textbooks studied in that 
grade, it was found that they give a statistically meaningless result due to the small number of cases (for example, 
in grade one only one task was found in the four textbooks). The proportions observed in the distribution of key 
concepts by task type (in one, two or three task types together) were also studied. 

Occurrence values of key concepts for all (N = 115) cases (textbook task: M = 0.93; SD = 0.24; Independent 
homework: M = 0.67; SD = 0.46; Task requiring Internet use: M = 0.23; SD = 0.42) (Cochrans’Q = 103.73; df = 2; p < 
.001; d = 6.064) indicated a significant and strong impact on the influence of task types. 

In the case of textbook tasks, distribution by grades seven, eight, nine, and ten showed a significant differ- 
ence (7?= 1.28; df= 3; p = .733; d= 0.230). The impact of grades on the occurrence of key concepts in these tasks 
is small (see Table 7). In the course of the examination by grades, key concept distributions in relation to textbook 
tasks that could be evaluated statistically were found only in grades seven, eight, nine, and ten. So, in this case, the 
number of evaluable cases is 97 (see Table 7). 


Table 7 
Relative Frequency of Key Concepts Related to Energy Awareness According to Task Types by Grades (N = 97) 


Textbook Independent Task requiring 


Grade task homework the use of the ae : df p d ae 
Internet 
7. 1.00 38 16 21.29 2 < .001 7.057 large 
8. .96 16 23 29.84 2 < .001 6.930 large 
9. 1.00 13 15 25.12 2 <.001 7.310 large 
10. 0.96 86 31 25.81 2 < .001 5.688 large 
0 1.28 13.00 2.02 
df 3 3 3 
p 733 .005 566 
0.230 0.782 0.291 
Effect size (d/) 
small intermediate small 


A significant difference in the distribution of key concepts according to the grades in Table 6 was found only 
in the case of independent homework (y= 13.00; df = 3; p =.005; d = 0.785), where grade ten had the highest pro- 
portion (M = 0.83) of independent homework related to the application of key concepts, followed by grades eight 
(M = 0.76) and nine (M = 0.73). The lowest rate was found in grade seven (M = 0.38). 

The proportion of textbook tasks for deepening and applying key concepts was significant and the highest in 
all grades followed by independent homework and then tasks requiring internet use. The proportion of the latter 
was significantly less compared to the other two (v?= 2.02; df= 3; p = .566; d= 0.291). 

When analyzing the key concepts from a didactic point of view, in the analysis by subject, the textbooks for 
all grades were used, as opposed to the examination of the distribution by grade, because the number of cases 
for each subject was evaluable (in the case of natural history subject, e.g., the textbooks for grades 1-6 were con- 
sidered together, which gave an evaluable number of cases). Therefore, here the number of cases of occurrence 
to work with was again N= 115. 
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Table 8 


Relative Frequency of Key Concepts Related to Energy Awareness According to Task Types by Subjects (N =115) 


Pyne Rate ian Geography Biology Physics + Chemistry x df p a 
Textbook task 60 .98 1.00 .96 1,00 20.96 4 < .001 0.959 Large 
Independent homework 40 67 55 87 85 13.72 4 .008 0.747 Intermediate 
Task requiring internet 30 18 44 19 42 4.63 4 327 0.415 Small 
Cochran's Q 1.55 59.91 6.00 39.46 5,20 
df 2 2 2 2 2 
p 459 < .001 < .001 < .001 074 
Effect size (c) 0.856 0.806 0.828 0.123 0.725 
large large large large large 


In the case of occurrence by subject, significant differences were found in the occurrence of key concepts 
by task type in the textbooks of geography, biology, and physics (see Table 8). Textbook tasks were found in the 
highest proportion followed by independent homework tasks, and by tasks requiring internet use in the case of 
all three subjects. Although there were no significant differences in the occurrence of key concepts by task type 
in the textbooks of natural history and chemistry subjects, a similar trend was observed in the distributions as in 
the case of the previous three subjects. 

Analysing each task type, the proportion of textbook tasks (y?= 20.96; df= 4; p <.001; d= 0.959) (see Table 8) 
was only smaller (M = 0.60) in the case of the natural history subject compared to the other subjects. Regarding 
independent homework (y?= 13.72; df= 4; p =.008; d= 0.747), physics (M = 0.87) and chemistry (M = 0.85) textbooks 
had the highest number of tasks compared to the textbooks of other subjects. The proportion of tasks requiring 
Internet use (y? = 4.63; df= 4; p =.327; d= 0.415) was small for all subjects. 

The distribution of key concepts by the three types of tasks was examined in the case of the didactic analysis 
as well. That is, whether the given key concept occurred in one, two, or all three of the textbook, homework, and 
Internet tasks. 

In the total sample (N =115), the key concept studied occurred only in one task type in 36 cases, in two task 
types in 61 cases and in three task types in 61 cases. In the case of co-occurrences, the effect of grades was found 
to be strong (y? = 25.29; df=11; p =.001; d= 1.080). In grades 1-6, single occurrence was dominant against double 
and triple occurrence (in grades 1-2 and 5-6 only single, in grade three single and double, in grade four single and 
triple), while in grades 7-12, co-occurrence in the three types of tasks was observed (double occurrence had the 
highest proportion, while triple occurrence had a smaller proportion). 

In the co-occurrence of key concepts according to didactic aspects (y? = 38. 68; df= 4; p =. 015; d=1.423) a 
significant difference was observed according to subjects and the subject effect was strong. In the case of each 
subject, single, double, and triple occurrences were found. In the case of natural history, the single occurrence (> 
80 %) was the most typical, and in the textbooks of the other subjects, the double occurrence was the most typi- 
cal. In summary, the occurrence of key concepts in the textbook tasks dominates both by grade and by subject, 
while tasks requiring internet use help the learning and application of key concepts related to energy awareness 
in the smallest proportion. 

In the following, it was examined whether a given key concept appeared in textbooks only as knowledge or 
as part of attitude shaping (question 2b). The values of the actual occurrences (N = 320) related to the total sample 
(Attitude shaping: M = 0.17; SD = 0.37; Knowledge: M = 0.99; SD = 0.08; Z = -15.52; p < .001; d =3.489) indicated a 
significant difference between the occurrences of the key concept at the level of factual knowledge and as part of 
attitude shaping, and the effect size was large. 
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Table 9 


Relative Frequency of Key Concepts Related to Energy Awareness According to Attitude-Shaping Aspects by Grades (N = 320) 


Grade Attitude shaping Knowledge Z p Effect size (d) 

ts 50 1.00 -1.00 < .001 2.000 large 
2. .00 1.00 -6.00 < 001 1.123 large 
3. 80 80 -0.00 1.000 0.000 no effect 
4. 1.00 1.00 -0.00 1.000 0.000 no effect 
5. 1.00 1.00 -0.00 1.000 0.000 no effect 
6. 92 1.00 -1.00 317 0.183 small 
7. 16 98 -7.28 < .001 3.891 large 
8. .06 1.00 -7.87 < .001 7.808 large 
9. .00 1.00 -7.00 < .001 9.123 large 
10. .05 1.00 -8.36 < .001 7.396 large 
ale 20 1.00 -2.00 046 2.828 large 
12. 00 1.00 -3.16 .002 10.665 large 
xv 12.73 15.08 
df 11 11 
p < 001 001 

Effect size (d/) 1.813 3.134 

large large 


The key concept distribution as part of attitude shaping according to grades (v7 = 12.73; df=11; p < .001; d 
= 1.813) indicated a significant and strong effect (see Table 8). The occurrence of key concepts as a textbook text 
for attitude shaping showed relative frequency values of .50 or higher in grades 4-5 (M= 1.00), six (M = 0.92), then 
three (M = 0.80) and one (M = 0.50). In other grades, this value was 0.20 or lower. 

The effect of grades on the occurrence of key concepts in textbook parts transferring knowledge (v7 = 15.08; 
df=11; p= .001; d= 3.134) was also significant and large. Relative frequency values of .80 or higher were observed 
in each grade. 

Regarding relative frequency values within grades, in grades three (M = 0.80), four and five (M = 1.00), key 
concepts occurred in equal proportions in textbook parts shaping knowledge and attitude. In all other grades, 
except for grade six (p = .317), the relative frequency values in textbook parts that transfer knowledge were sig- 
nificantly higher. 

The key concept distribution as part of attitude shaping by subjects (y? = 157.40; df= 4; p < .001; d= 1.967) 
indicated a significant and large effect (see Table 9). The relative frequency of key concepts in the attitude-shaping 
text was greatest in the textbooks of the natural history subject, while relative frequency values in the case of the 
rest of the subjects were 0.21 or smaller. 


Table 10 
Relative Frequency of Key Concepts Related to Energy Awareness According to the Aspects of Attitude Shaping by Subject 
(N= 320) 
Attitude shaping Natural 
aspects history Geography — Biology Physics Chemistry va df p Effect size (d) 
Attitude shaping 91 05 16 03 21 157.40 4 < .001 1.967 Large 
Knowledge 97 99 1.00 1.00 1.00 3.36 4 499 0.206 Small 
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Attitude shaping Netural 
aspects history Geography —_ Biology Physics Chemistry x df p Effect size (d) 


Z -1.00 -12.04 -4.58 -7.28 -4.69 
p 317 <.001 < 001 <.001 < 001 
0.348 7.248 4.567 9.291 3.826 

Effect size (d) 
Small Large Large Large Large 


Relative frequency in information-providing textbook sections by subject (y? = 3.36; df= 4; p =.499; d= 0.206) 
showed no significant difference, the effect of the subjects in this case was small. Relative frequency values in the 
textbooks of all subjects were 0.97 or greater. 

In the case of all subjects, key concept relative frequencies were significantly greater in textbook parts that 
provide knowledge except for the natural history subject in which there was no significant difference (p = .317) 
between occurrences in attitude-shaping and information-providing textbook parts. Overall, the vast majority 
of the textbooks studied discuss the studied key concepts at the factual knowledge level compared to textbook 
sections related to attitude shaping. 

With regard to the aspects of attitude shaping, the examination of joint occurrences in the attitude-shaping 
and information-providing sections was also carried out. We examined whether the given key concept occurs only 
in one or both types of textbook material. Considering the total number of cases studied (N = 320), there were 261 
single and 59 double occurrences. 

There was a significant difference between double and single occurrences by grades in the total sample 
(N=320) (v7 = 165.08 df= 11; p < .001; d= 2.064) and the effect of grades here was strong. In grades two and 7-12, 
a single occurrence was significantly characteristic, primarily as information-providing parts (> 85%). In grade one, 
the attitude-shaping and information-providing sections regarding the key concepts occurred in equal propor- 
tions (= 50%), while in grades 3-6, co-occurrence in attitude-shaping and information-providing sections was 
characteristic (double occurrence: > 60%). 

Examining single and double key concept occurrences by subject (vy? = 145.69; df= 4; p <.001; d= 1.828) there 
was also a significant difference between them, and a large effect of the subjects was detected. In natural history 
textbooks, the double occurrence ( > 80%) was characteristic, and in all other subjects, the single occurrence of 
key concepts ( > 78%), primarily as an information-providing textbook section was typical. 


Content Analysis from Structural Aspect 


In the course of the content analysis from the structural aspect, the differences in the occurrence of key con- 
cepts were examined in the core material, readings, complementary text of the textbooks or in the form of a figure 
or image with the evaluation of the distribution of key concepts among the structural units of textbooks (Research 
question 3). Studied cases concerned here again only the actual occurrences (N = 320). This analysis was pointless 
in those textbooks where the concept did not occur. 

Considering the total number of cases (N = 320), the occurrence of key concepts in the core material, reading, 
complementary text, or in the form of a figure or image (in core material: M = 0.84; SD = 0.35; in reading: M = 0.09; 
SD = 0.29; in complementary text: M = 0.09; SD = 0.28; figure, picture: M = 0.27; SD = 0.44) (Cochrans’Q = 270.07; df 
= 3; p < .001; d= 4.561) showed a significant difference verifying a large effect. 

Examining key concept occurrences in the core material by grade (7? = 26.17; df = 11; p =.006; d= 0.593) a 
significant difference was found indicating an intermediate effect of the grades (see Table 10). The occurrence of 
key concepts in the main text by grade shows relative frequencies greater than 0.50 in grades apart from 1-2 and 
three and eleven (see Table 11). In grade three, no key concept was encountered in the core material. 
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Table 11 


Relative Frequency of Key Concepts Related to Energy Awareness According to the Aspects of Structure Analysis by Grade (N= 320) 


Grade Main text Reading a ee a df p a 

1, 50 .00 .00 50 2.00 3 572 2.824 large 
2. 50 .00 .00 50 2.00 3 572 1.705 large 
3. .00 .00 .00 1.00 8,00 3 <.001 4.896 large 
4. 1.00 50 25 25 5,14 3 .162 3.322 large 
5. 80 07 33 46 1.02 3 .001 3.867 large 
6. 1.00 07 .00 53 33.69 3 <.001 4.063 large 
7. 19 MM 17 13 85.32 3 <.001 3.568 large 
8. .90 07 05 24 125.90 3 <.001 3.322 large 
9. 76 22 .08 45 48.39 3 <.001 17.426 large 
10. 92 .00 03 20 161.08 3 < .001 2.260 large 
11. 40 40 40 60 0.529 3 912 0.685 intermediate 
12. 90 10 .00 30 17.71 3 .001 5.603 large 
V7 26.17 35.89 30.76 31.56 
df 11 11 11 11 
p .006 <.001 001 001 

Effect size 0.593 0.710 0.648 0.659 
(0) intermediate interme- intermediate interme- 

diate diate 


Key concept occurrences in readings (y? = 35.89; df = 11; p < .001; d= 0.710) also showed significant differ- 
ences between grades and the effect of the grade here was also intermediate. In grades 1-3 and ten, the studied 
key concepts were not encountered in readings. Relative frequency values in readings were greatest in grade four 
(M = 0.50), while all other grades had relative frequency values of 0.40 or lower. 

In the complementary text again, significant differences could be detected in the relative frequency values 
of the key concepts by grade (y? = 30.76; df = 11; p =.001; d = 0.648) with intermediate effect size. There was only 
one outlier relative frequency value in grade eleven, but it was not high either (M = 0.40). In grades 1-3, no key 
concept was present in complementary texts. In the other grades, relative frequency values of 0.33 (fifth grade) 
or lower were detected. 

In the case of occurrence in the form of figures or pictures, there were significant differences by grade (y? = 
31.56; df = 11; p = .001; d= 0.659). The effect of grades, in this case, was also intermediate. Interestingly, in grade 
three, key concepts were encountered only in the form of figures, pictures. The relative frequency value was the 
highest in grade eleven (M = 0.60), followed by grade six (M = 0.53) and grades 1-2 (M = 0.50). In other grades, 
relative frequency values lower than 0.50 were found. 

Considering analysis within the grades, in grades 1-3, the occurrence of key concepts in the main text and in 
the form of a figure or image was typical (in grade three, only figure and picture occurrence could be observed). 
In the other grades, the occurrence in the main text dominated, compared to the occurrence in reading, comple- 
mentary text, figures, and pictures. Grade eleven was the exception, where the occurrence in figures and pictures 
dominated (M = 0.60), the other three had relative frequency values of 0.40. 

The occurrence of key concepts in the main text showed no significant difference by subject (y? = 4.42; df= 
4; p =.351; d= 0.238). The impact of the subjects, in this case, was small (see Table 11). The highest proportion of 
occurrences in the main text was detected in the natural history textbooks (M = 0.90), but for the other subjects 
the relative frequency value was less than 0.70 (see Table 12). 
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Table 12 


Relative Frequency of Key Concepts Related to Energy Awareness According to the Aspects of Structure Analysis by Subject 
(N= 320) 


Structural aspects fae Geography Biology Physics Chemistry xr df p Effect size (d) 
Main text 90 0.82 0.80 0.81 0.70 4.42 4 351 0.238 small 
Reading 10 0.04 0.04 0.29 0.13 2713 4 <.001 0.614 intermediate 
Complementary text 13 0.07 0.08 0.07 0.23 7.85 4 097 0.319 small 
Figure, picture 45 0.25 0.20 0.38 0.09 15.39 4 .004 0.453 small 
Cochran's Q 6.33 156.34 36.94 61.31 2.65 
df 3 3 3 3 3 
p < .001 < .001 <.001 < 001 < .001 
Effect size (d) 3.234 7.691 2.231 3.345 7.213 
large large large large large 


There was a significant difference in the distribution of key concepts occurring in readings by subject (vy? = 
27.13; df= 4; p < .001; d= 0.614), the subject effect is intermediate. Relative frequency was the highest in the case 
of physics (M = 0.29), in the readings of the textbooks of other subjects this value was 0.13 (chemistry) or lower. 

There was no significant difference in the occurrence of key concepts in complementary texts by subject and 
the subject effect was small (y? = 7.85; df = 4; p = .097; d = 0.319). Key concepts occurred most frequently in the 
complementary texts of chemistry textbooks (M = 0.23). In the case of other subjects, this occurrence showed a 
relative frequency of 0.13 (natural history) or smaller. 

The difference between the relative frequencies of key concepts in the form of figures and pictures by the 
subject was significant, but the subject effect was small (y? = 15.39; df= 4; p =.004; d= 0.453). Key concepts occurred 
in figures and pictures most frequently in natural history (M = .45), and physics textbooks (M = 0.38), while in the 
case of other subjects, relative frequency values of 0.25 (geography) or lower were detected. 

Key concepts occurred significantly in the greatest proportion in the main text in all subjects, followed by 
figures, pictures, and then in readings and complementary text. Overall, both the analysis by grade and by subject 
revealed that most of the knowledge related to EA occurred in the main text which contained the curriculum re- 
quirements, and then in the accompanying figures and pictures. The occurrences in readings and in complementary 
texts were smaller compared to the previous two. 

Co-occurrence was studied in the course of the structural analysis as well. The relative frequency of key con- 
cepts was analysed in one, two, three, or all four structural elements, i.e., the main text, readings, complementary 
texts, figures, and pictures. Considering the total number of cases (N = 320), 231 single, 64 double, 11 triple and 
4 quadruple occurrences were found. 

Studying the differences in co-occurrence by grade in the structural analysis (y?= 111. 91; df= 11; p <.001;d 
= 1.459) significant differences were found. The effect of grades on co-occurrence was large. Single occurrences 
were not found only in grades 2-3. The double occurrence was also absent in two grades, in the textbooks of grades 
one and three. The absence of triple occurrence was detected in grades one, two, seven and ten. The quadruple 
occurrence was only found in grades 7-9. 

Considering the structural analysis, significant differences were found in the valid per cent values in the dis- 
tribution of the co-occurrence of key concepts by subject (y?= 40.32; df= 4; p = .001; d= 0.800), and the subject 
effect was large. In the textbooks of all subjects, a single occurrence was dominant (in the main text), followed by 
double, triple, and quadruple occurrences. In the textbooks of geography and physics, all cases were detected, 
while in biology and chemistry, only single and double occurrences were detected. In natural history, only the 
quadruple occurrence was not found. 
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Discussion 


The aim of the study is to present textbook research that is now one of the most topical environmental edu- 
cation tasks in the period of the global energy crisis, exploring the potential of educating students on EA in 67 
Hungarian primary and secondary science textbooks (grades 1-12, 5 subjects: natural history, geography, biology, 
physics, and chemistry). In the field of environmental education, including EA education, as part of the attitude, it is 
important what kind of energy awareness knowledge students have, whether they can behave energy-consciously, 
and what emotions and beliefs they have about EA. The question is to what extent textbooks, including textbooks 
on science subjects, can enhance the threefold function of EA education mentioned here. 

Textbooks are important aids of learning, which reflect and thus provide an opportunity to achieve the learn- 
ing and teaching goals set out in the curriculum requirements (including the development of various student 
competencies) through the communication of the teacher. Consequently, their content, structure and pedagogical 
characteristics have a fundamental impact on the process of teaching and learning, in terms of the knowledge, 
abilities and attitudes to be developed (Chambliss & Calfee, 1998). The textbook analysis model, which emphasizes 
the constructive learning process and was chosen as the theoretical basis for our textbook research, devotes an 
important role to textbooks in the shaping of the knowledge, abilities and attitudes created as a result of learn- 
ing. According to the constructive model, one of the main aspects of textbook analysis is the extent to which the 
textbook helps to provide a student-centred teaching. For this, it is essential that the content of the textbook is 
understandable, concise, motivating for the student, contains tasks for deepening and applying knowledge, builds 
on the emotional factors of the students, presents the relationship between the knowledge acquired in school 
and everyday life, and encourages independent learning. They are all textbook functions, if implemented, the 
development of an attitude towards EA can also be ensured. 

As a meeting point between the main aspects of the textbook analysis model with constructive learning in 
mind (knowledge, ability and attitude) and the individual elements of the attitude towards EA (knowledge about 
EA, behaviour, emotions), quantitative conceptual content analysis was applied in three dimensions: 1) Professional 
content dimension: in which first the occurrence of the key concepts and knowledge related to energy awareness 
specified in the curriculum requirements (‘renewable energy; ‘energy saving; ‘heating; ‘power station’ ‘conscious 
energy use; ‘energy crisis, ‘energy efficiency, and ‘energy consumption’) was analysed in the studied science text- 
books. If these concepts occur in the textbooks, in what ways do they help students to understand and see the 
essence? Whether they occur in the form of short definitions or circumscribed text containing descriptions, or 
explanations. In the case of the occurring key concepts, their distribution between these forms was also studied. 
2) In the course of the content analysis from the pedagogical aspect, first the typical elements for the deepening 
and application of key concepts related to EA were identified in the studied textbooks. For completing this didactic 
task, studying textbook tasks, independent homework tasks and tasks requiring the use of the Internet seemed 
suitable. Thus, the distribution of key concepts related to EA according to these tasks was analysed here as well. In 
the second part of the pedagogical analysis, the occurrence of key concepts clearly related to EA was examined 
in the attitude shaping (behaviour, emotions, beliefs, attitude shaping) texts. It was examined whether the key 
concept occurred as a knowledge or attitude element. 3) In the course of the content analysis from the structural 
aspect, it was assumed that it was not irrelevant for a child, regarding learning, that the knowledge to be learned 
occurred in the main text containing the teaching material, or in motivating readings containing curiosities, or 
in images in figures that provide additional information, help understanding or motivate. Therefore, the relative 
frequency of the key concepts related to EA in the main textbook text, readings, supplementary texts and figures, 
images was analysed here. 

The occurrence of key concepts was examined first in the total sample in all dimensions, regardless of grades 
and subjects. Then occurrences were analysed by grade and subject. Finally, it was examined in each dimension 
that the key concepts occurred in how many of them. The reason for the latter analysis was that the more the given 
knowledge occurred, the deeper the created knowledge of the students about it would be. 

In the course of the content analysis from the professional aspect, it was found that key concepts related to EA 
were not present at all in 18 of the 67 textbooks studied. This may also be explained by the fact that the concepts 
and knowledge studied were not related to the content of the given textbooks. The concepts of ‘renewable energy’, 
‘energy crisis, and ‘power station’ occurred in the largest proportion in the total sample (N= 62, including multiple 
occurrences of a given key concept in different aspects in different topics of a given textbook) followed by the 
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concepts of ‘heating; ‘energy consumption; ‘conscious energy use; and then ‘energy saving’ and ‘energy efficiency’ 
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Breaking down the occurrence of key concepts by grade, two large groups could be distinguished. One group 
included grades 1-4, where zero or small relative frequency values were typical, and the other group included 
grades 5-12, where these values were much greater. In grades 1-4, the concepts of ‘renewable energy; ‘energy 
saving; ‘heating; and ‘power station’ were encountered, and then, towards grade 12 more and more key concepts 
related to EA occurred. In grades 1-4, the concepts that actually occurred were described mainly in a circumscribed 
form. In many cases, it was not even the key concept itself that was included in these descriptions, but another 
knowledge or concept within the scope of the key concept (e.g., the concept of ‘renewable energy’ in the text- 
books for grades 1-2 grade occurred as wind or water, which we can use to generate electricity as well). These were 
correctly conceptualized at an age-appropriate level of intellectual development, mostly in a childish language. 
However, from grade five increasingly abstract, short, definitions of key concepts were found in greater propor- 
tions, which can make their understanding and learning difficult (Adamik et al. 2004; Nagy & Varga, 2014), if there 
was no explanatory text, picture, figure, task, or teaching method to help understanding. This entails maintaining 
a definitive knowledge at a factual knowledge level in most grades, which is not favourable for the development 
of behaviour and emotional elements related to EA. 

When studying the occurrence of key concepts by subject, it was found in the total sample (N=621) that the 
key concepts occur in all subjects except for the concepts of ‘energy crisis’ and ‘energy efficiency’ The concept and 
knowledge of ‘energy crisis’ were absent from the textbooks of the natural history subject, while ‘energy efficiency’ 
was missing from the textbooks of physics. In line with previous studies (Konya, 2018b; Revakné, 2018), geography 
textbooks are dominant in terms of occurrences, followed by physics, chemistry, and biology. This result should be 
treated with care. In this case, it has to be taken into account that the number of topics to which EA can be related 
is different in the given subject textbooks (Konya, 2018). Furthermore, the question is whether textbooks always 
take advantage of mentioning and teaching the aspects of EA when discussing a given topic. A deeper exploration 
of this could be the subject of another research. Considering the description of the occurring concepts, a similar 
result to those of the grades can be seen. That is, definitions played a major role in the textbooks of all subjects, 
which may result in that information getting stuck at the factual knowledge level regarding EA attitude shaping. 

When the occurrence of key concepts was analysed, it was revealed that a single occurrence predominates 
(307 cases were either definition or circumscription and 13 cases were occurrences in both modes of description). 
Key concepts found in similar proportions in one or both forms were typical only in grades 1-3. The vast majority 
of single occurrences (grades 5-12) were in the form of definitions, which, in line with previous research (Coker et 
al., 2010), suggests a rather definitive nature of knowledge related to EA in the textbooks studied. 

In the didactic relations of the content analysis from the pedagogical aspect, the proportion of concept 
distribution between textbook tasks, independent homework and tasks requiring the use of the Internet was ex- 
amined in the case of actual key concept occurrences. Considering the total sample (only N=115 due to the actual 
occurrences in the tasks), in most cases the knowledge related to EA was deepened and applied in the textbook 
tasks, followed by the independent homework tasks, and finally in the tasks requiring internet use. The analysis 
by grade and subject yielded similar results. If we consider the fact that only 115 of the actual 320 occurrences in 
the 67 textbooks were observed in these tasks, this represents almost a third of the deepening and application of 
the key concepts that actually occur related to EA. Again, it should be taken into account that the content related 
to EA makes what proportion of the content of that subject. Regarding the textbook occurrence of key concepts, 
this one-third application is small and does not help to develop the attitude related to EA. Nor is it the case that 
only grades 7-10 had a key concept in each of the three types of tasks, and in the other grades only one of the 
three. Further, deeper analysis is required to identify the form of work (e.g., cooperative work, project, etc.), method 
(e.g., observation, experiment), thinking operation (e.g., problem solving, inductive thinking, divergent thinking, 
etc.) required by the solution of the above task types, all of which can be important components of EA education. 

In the section on attitude shaping in the content analysis from the pedagogical aspect, in accordance with 
previous research results (Chao.et al., 2017; Revakné et al. 2018), it was found that the vast majority of textbooks 
provided factual knowledge related to EA both in terms of all actual key concept occurrences and in terms of 
analysis by grade and subject. This is certainly a fact to consider in the future for textbook writers, partly because 
they only partially meet the requirements of the curriculum related to EA in terms of attitude development, and 
also because the development of a correct attitude related to EA is thus helped less by textbooks. 

The content analysis from the structural aspect focused on how the occurring key concepts were divided 
between the main text of textbooks, readings, complementary text and figures, and pictures. The examination of 
these four aspects of structural analysis was partly justified, on the one hand, by the fact that these were typically 
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the structural units that occurred almost everywhere in the structure of textbooks, and on the other hand, by the 
assumption that it does matter in what structural unit a particular knowledge occurred. The main text is the unit ina 
textbook which contains the most important information and curriculum requirements for learning, both students 
and teachers pay more attention to it, learn from it, and teach it. Apart from the main text students mainly focus 
on the explanatory, information-providing and motivating figures and images belonging to the main text. It is a 
common experience that students and their teachers often overlook readings and complementary texts, despite 
the fact that they often contain interesting information, not considering it important. However, they sometimes 
provide knowledge and complementary information to the content of the main text that help to understand it, give 
examples of the connection of the learned knowledge with everyday life, its practical application or even contain 
attitude-shaping elements. They would therefore be important in the development of applicable knowledge and 
in the shaping of attitude as well. 

Present research confirms this experience. Most of the studied knowledge related to EA occurred in the main 
text, and then in the accompanying diagrams and images. The occurrence rate in readings and complementary texts 
was smaller compared to the previous two. However, regarding this result, it should be noted that in textbooks the 
ratio of readings and complementary texts is smaller compared to the main text, figures, and images. Not all main 
texts have readings or complementary texts. When examining multiple occurrences in structural units, it was also 
detected that in the textbooks of different subjects, a single occurrence was typical, predominantly in the main text. 


Conclusions and Implications 


This study presented the analysis of natural sciences textbooks in relation to energy awareness. The analysis was 
based on the content analysis method which bears the constructive learning process in mind. The applied method 
was quantitative content analysis. In the course of the conceptual content analysis, the occurrence frequency of 
the key concepts related to EA was examined from professional, pedagogical and structural aspects. The results 
show that concepts related to ER are present in the textbooks studied. Most of the concepts occur in definition- 
like texts and circumscriptions which better facilitate understanding are less frequent. Textbook elements shaping 
attitude and behaviour are less frequent than factual knowledge. Thus, natural sciences textbooks to be edited 
in the future shall include more explanations and descriptions better helping and motivating the understanding 
of concepts related to EA. It would be worth devoting separate lessons, pages to this topic. An increase in the 
volume of texts, visual elements, tasks (tasks requiring activities, observation, student examinations related to EA, 
modelling, projects, collaborations studying everyday experience, tasks facilitating ICT and processing internet 
sources) shaping attitude and behaviour related to EA shall occur. It will still be a question, of how teachers would 
transfer the content of such corrected textbooks towards the students. As this fundamentally determines how 
the textbook content transforms into knowledge and how attitude and behaviour related to EA are shaped, and 
what role textbooks can take in shaping knowledge and attitude compared to other school and social factors. This 
could be the topic of future research. 
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Introduction 


Concepts are the foundation of human knowledge structures (Johnson 
et al., 1994), which have been the center of attention in linguistic psychology 
(Taber, 2013). In recent years, conceptual research has received increasing 
attention in education. 

Drawing from Bruner’s structure curriculum theory, students’ construc- 
tion and possession of an organized discipline knowledge system is an es- 
sential aim of science education, including chemistry (Bruner, 1964). Prior to 
this, students were required to obtain an understanding of the basic chemistry 
concepts and principles (Bruner, 1964; Pinar, 2004). Many studies have sug- 
gested that a lack of understanding of chemical concepts impedes students’ 
abilities to solve chemical problems and damages the integrity power of the 
whole conceptual structure (Burrows & Reid Mooring, 2014; Wang & Barrow, 
2013). However, as these discipline concepts are not part of our daily lives, 
students do not have sufficient preconceptions to make connections with 
these abstract and complicated concepts. Thus, it is challenging for students 
to assimilate new concepts into their minds to establish connections with 
their existing conceptual structure (Joki et al., 2015). As one of the Science, 
Technology, Engineering, and Math (STEM) disciplines, chemistry contains 
many basic abstract concepts that are essential for the body of knowledge. 
Therefore, exploring chemical concepts and understanding their relation- 
ships is crucial. 

Many new concepts have emerged during the development of Chemis- 
try and the reform of curriculum standards, which indicates the importance 
of exploring chemical concepts and the structure of chemistry (Apriwanda 
et al., 2021; Treagust et al., 2000). Specifically, chemistry has evolved from 
macroscope to microscope levels, from simple to complex molecules, from 
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Abstract. A solid grasp of basic concepts 
and their relationships can stimulate 
learning motivation, improve teaching 
quality, and help students to build a body 
of knowledge in a discipline. Most of the 
previous studies have considered one 
chemistry topic amongst many others. 

To date, no studies have been conducted 
to explore the conceptual structures of 
teachers and students based on chemical 
concepts. Experts, teachers, and students 
(N=2348) from three public universities 
and ten upper-secondary schools in China 
participated in the studies. In Study 1, 
personal interviews, material evaluation, 
and importance rating questionnaires 
were conducted to determine the core 
concepts. In Study 2, the multidimensional 
scaling (MDS) and cluster analysis (CA) 
were conducted to explore the conceptual 
structures of upper-secondary school 
teachers and students. Then the structural 
differences between high- and low- 
achieving students were compared. The 
results showed that the three-dimensional 
solutions were appropriate for the 
conceptual structures of teachers and 
students, respectively. The high-achieving 
students have a more scientific and 
organized conceptual structure than those 
of low-achieving students. This study is 
the first to explore core concepts and their 
structure with a large sample size, and the 
reliable results help students understand 
abstract chemical concepts and improve 
their interest in scientific disciplines in the 
future. 
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static to dynamic perspectives, and from one single discipline to inter-discipline. During this transition, students 
and teachers update their concepts, reorganizing them as new concepts. For instance, inductive effect and refuse 
classification emerged in the new chemistry textbooks. Besides, China releases the revised version of Chemistry 
Curriculum Standards for Compulsory Education in 2022. Implementation of these standards indicates the accom- 
plishment of the construction of chemistry knowledge based on the big ideas and core concepts. These standards 
also stressed the importance of developing students’ integrated understanding of the big ideas and core concepts 
in scientific literacy (Stern & Roseman, 2004). 

Considering the importance of understanding concepts for chemistry learning, it is necessary to determine 
core concepts and explore their structure in the context of disciplinary development reforms. Since core concepts 
are the most important part of the concepts pool, chemistry teachers can use these core concepts as a general 
reference and implement them in future teaching process. 

For the conceptual structure representation, multidimensional scaling (MDS) has been widely used due to its 
simplicity and scientific objectivity. This method visualizes the representation of the global relationships among 
concepts through the participants’ categorical data of concepts (Arce et al., 2010; Mai et al., 2021). The specific steps 
of this method are listed as follows: (a) classifying concepts, getting distance data, and resenting these distance 
data as relationships among research objects in a multidimensional space; (b) obtaining the fit indexes, spatial 
coordinates, and the distribution map of each dimension solution; and (c) determining the optimal dimension solu- 
tion by the fit indexes, then analyzing the distribution map of the optimal dimension (Gordon & Hayward, 1973). 

Although the distance data obtained from the participants’ classification of concepts are objective, the trans- 
formation from data into a distribution map and the judgement of the conceptual structure based on the distribu- 
tion map is subjective and is affected by knowledge, experience, or cognitive ability. In other words, individuals 
with abundant chemical knowledge or a high level of cognitive ability are able to summarize the commonalities 
of some concepts in the map well and are not confused by the misclassification of a concept in the map. Therefore, 
further stacking using cluster analysis (CA) methods is needed to make the classification results more objective 
and precise (Mai et al., 2021; Qian et al., 2009). The dimensional coordinates of each concept can be obtained using 
CA, which significantly reduces the difficulty or inconsistency of clustering according to the spatial distribution 
map (Huang & Qian., 2020). 

To date, a handful of studies have explored concrete chemical concepts in some topics, such as acid-base 
balance structure (Wilson, 1998), chemical equilibrium (Mai et al., 2021), and organic chemistry (Hrin et al., 2018). 
However, no study was conducted to explore the core concepts and conceptual structures of upper-secondary 
teachers and students in the field of chemistry discipline but not only in one specific topic. This study aims to ad- 
dress this gap. 

According to the previous discussion, the present research aims to answer three questions: 

Q1. What are the core concepts in the perception of chemistry by teachers and students? 

Q2. What are the features of conceptual structures in chemistry for teachers and students? 

Q3. Are there any differences in conceptual structures in chemistry between high- and low-achieving students? 


Research Methodology 
General Background 


Two studies were conducted to determine the chemistry core concepts and teachers’ and students’ concep- 
tual structures. In Study 1, the core concepts were determined by personal interviews, material evaluation, and an 
importance rating questionnaire. 

In Study 2, MDS and CA methods were used to explore the conceptual structures in upper-secondary school 
teachers and students. First, a free classification analysis was used to classify the core concepts by teachers and 
students. Then MDS analysis was then performed using the free classification data to determine the structural 
dimensions of these core concepts. Second, the CA was conducted to obtain the multidimensional representa- 
tions data, which were analyzed to classify the core concepts. Finally, the differences in the conceptual structures 
between high- and low-achieving students were compared. 

Overall, the results of the present studies would contribute to the establishment of the chemistry core con- 
cepts and provide suggestions for the design of chemistry teaching materials—a theoretical basis and practical 
guidance under the background of the new chemistry curriculum reform in China. 
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Participants 


Two thousand four hundred experts, teachers, and students from three public universities and ten upper- 
secondary schools in nine areas of China were selected by convenient sampling. Excluding 52 invalid questionnaires 
(efficient rate=93.92%), the final sample of participants was 2348. Table 1 shows the distribution and subjects in 
the two studies (see below for the description of the Instrument and Procedures). 


Table 1 
Demographic Characteristics of Participants 


Study Group n 
Group 1. Chemistry experts and chemical educators 12 
Group 2. Undergraduates (55) and postgraduates (31) 86 
Study 1 
Group 3. Upper-secondary chemistry teachers 405 
Group 4. 12th-grade upper-secondary school students 1109 
Group 5. Upper-secondary chemistry teachers 346 
Study 2 Group 6. High-achieving students in chemistry 197 
Group 7. Low-achieving students in chemistry 193 
Total valid participants (N) 2348 


Instrument and Procedures (Study 1) 


Study 1 had two steps, concept extraction and concept selecting. The first step includes personal interviews 
and material evaluation; the second step refined the concepts extracted in the first step through an importance 
rating questionnaire (processes are shown in Figure 1). 

For one-to-one interview, the chemists and chemical educators were interviewed separately. Interviewees 
were informed in advance of the time, place, and topics of the interview. 

We adopted a semi-interview that contained three questions: 

1. From the perspective of the development of chemistry, what are the core concepts? (Major question). 

2. What is your main research field? (Minor question). 

3. What are the concepts in frontiers of contemporary Chemistry?” (Minor question). 

4. Allinterview records were transcribed into texts. Repetitive concepts were removed and summarized 
in an Excel sheet. 

In addition to the interview, material evaluations were conducted to extract the concepts. Eighty-six under- 
graduate and postgraduate students, majoring in Subject Teaching (Chemistry) and Chemistry Curriculum and 
Pedagogy, were engaged in material evaluation. These materials stem from three parts: chemistry tests in the col- 
lege entrance examination, the chemistry curriculum standards, and upper-secondary school chemistry textbooks, 
all covering concepts in the upper-secondary school chemistry curriculum. Students were asked to summarize the 
concepts appeal in the materials in Excel sheet and count the frequency of occurrence of each concept. 

After combining two parts of chemical concepts and removing repetitions, the rest were randomly arranged 
to form an importance rating questionnaire. Frontline chemistry teachers and 12th-grade students were asked to 
assess the importance of each concept on a 7-point Likert scale (1=least important, 7=extremely important). For 
these participants, the frontline teachers were all in-service teachers with more than one year of teaching experi- 
ence; the 12th grade students had taken the elective chemistry course and reviewed the chemistry knowledge in 
the first round of review at upper-secondary school, so the structure of chemistry knowledge was formed in their 
perceptions. The higher the score of each concept, the more important the concept is in the upper-secondary 
school chemistry curriculum. Finally, the concepts with high scores were selected as the core concept. 
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Figure 1 


Instrument and Procedures (Study 2) 


Study1 


Step 1 
Personal 
interview 
(Group 1) 


Importance / CA 
+ rating (Group 5, 6, (Group 5, 6, 
(Group 3, 4) and 7) and 7) 


Materials 
evaluation 
(Group 2) 


The one hundred core concepts were formed in Study 1 and their structure further validated in Study 2, which 
also had two steps. Before Study 2, the top 25% and bottom 25% of students were identified based on their chem- 
istry scores in the college entrance examination (the first mock chemistry examination) and defined two groups 
as high- and low-achieving students, respectively. 

In step 1, upper-secondary school chemistry teachers and two group students were asked to freely classify 
the 100 core concepts in a paper questionnaire. Classification data were manually entered into digital data. The 
entering process was completed in approximately three months. The digital data were superimposed to a dissimi- 
larity matrix of 100x100 by the Style program. Following the rules, the matrix intersection of the concepts in the 
same category is marked as 0 and marked as 1 otherwise. The classification data of all participants were imported 
into SPSS software. The organization structure was explored using MDS. The fit indexes of each dimension and the 
spatial coordinate of each core concepts were obtained. 

The optimal dimension was selected by fit indexes. The spatial coordinates were used for the cluster analysis 
(CA) in step 2. Finally, the differences in the conceptual structures between high- and low-achieving students were 
compared. 


Data Analysis 


The preliminary analysis (mean scores and standard deviations) of each core concept, MDS, and CA were 
conducted using IBM SPSS (version 25, USA). 


Research Results 
Study 7 


The interviews of chemists and chemical educators revealed that the concepts they mentioned are broad 
and forefront. There are classic chemical concepts (e.g., molecule), cutting-edge interdisciplinary concepts (e.g., 
nano-structure), and practice technical concepts (e.g., simulation experiment). After excluding the repetition, 167 
concepts were finally listed. For evaluating materials, three parts of materials were combined before excluding the 
repetition, and finally ended up with 214 concepts. 

After combining two parts of chemical concepts, removing repetitions, and keeping the rest in random order, 
an importance rating questionnaire was formed. Using the SPSS, the mean score of the importance of each concept 
ranges from 2.15 to 6.42. Using the median score (i.e., 4) of the importance of concepts as the selection criterion, 
96 concepts with an importance score of 4 or higher were selected out. Additionally, with reference to chemistry 
textbooks and the opinions of chemists and school teachers, four concepts were added as concepts because of 
their high frequency in China’s college entrance examination after removing the concepts of college chemistry: 
rustiness (3.94), slow oxidation (3.93), chemical energy (3.92) and alloy (3.69), which were added as concepts with 
an average importance score close to four. Finally, 100 chemical concepts were selected just right (see Table 2). 
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Table 2 


The One Hundred Core Concepts in Chemistry 


Core 


n  CoreConcepts M n Core Concepts M n Core Concepts M M 
Concepts 
1 petiodictable 6.42. 26“ Tassifaclinotthe 5.69 54 molecule 528 76 homologue 4.71 
g leworoonseNe” gap. oy reduction 558 52  electrontransfer 5.26 -«77_~—«Saturatedydro- 4&4 
tion of mass . . e carbon . 

cee eC y ol ane cme ean cull ees isotope 5.20 78 combustion 4.67 
elements rium 

4 redox reaction 6.33 29 electrolyte 5.56 54 nuclear charge 5.20 79 proton 4.66 

5 chemical equation 6.29 30 ion reaction 5.56 55  decompositionreaction 5.19 80  organicreaction 4.58 

gp. “aineuntctsub. Gay ay. “MOND Feplacemenl 55: 366 electrolysis 517 81 atomicnucleus 4.51 

stance reaction 
a mole 6.22 32 solution Ce yf functional group 5.14 82  greenchemistry 4.43 
8 molarity 6.21 33 nouialzawoht pea Be ionic bond 5.13 83 voltaic cell 4.43 
reaction 

9 molar mass 6.16 34 compound 5.50 59 chemical bond 5.08 84 weakelectrolyte 4.37 

10 pH 6.06 35 ionization 5.48 60 composition 5.02 85 compose 4.35 

11. chemical property 6.03 36 atom 5.47 61 organic compound 499 86 eae 4.31 

12 elementalsymbol 5.91 37  acid-baseindicator 543 62 material structure 4.98 87 atomic group 4.29 

13 ionic equation 5.88 38 saturated solution 5.42 63 rates of reaction 495 88 acid rain 4.28 

14 Leal 5.80 39  atomicstructure 5.42 64 covalent bond 4.95 89  Tyndalleffect 4.28 

15 element 5.80 40 substance property 5.42 65 catalyst 4.91 90 particle 4.22 

16 chemicalreaction 5.78 41 element 5.80 66 noble-gas configuration 4.90 91  chemicalbattery 4.20 

eeueh [eee average molecular reenhouse 

17 extra-nuclear «5.7342 hydroxide 540 67 eae 480 oo eee, 4.15 
electron 9 

18 valence 5.71 43 electron 5.36 68 valence state 484 93 crystallization 4.15 

19 chemicalformula 5.69 44 isomerism 5.34 69 solute 480 94 energy 4.07 

single displace- : : 

20 meant reaction 5.67 45 hydrolysis 5.34 70 flame test 4.78 95 colloid 4.06 

21 pure substance 5.65 46 ion Be 7A. physical change 476 96 crystal 4.01 

a I 568. ar oxide 531 72 mixture 474 97 — tustiness 3.94 

23 nlnatCn e546 solubility 531 73 peroxide 4.74 98 — slowoxidation 3.93 
reaction 

24 atomic number 5.63 49 substitution reaction 5.29 74 Classification of sub- 473 gg chemical energy 3.92 

25 chetilcalequr 5.62 50 addiionreaction 5.2975 solvent 473 100 alloy 3.69 
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Study 2 


The optimally fitting model was selected on the basis of indexes of Stress and RSQ. Stress values between 
0.05 and 0.10, and RSQ values close to 1 indicate acceptable fits (Borg & Groenen, 2006). The results shown that 
the fit indices of three-dimensional solutions are appropriate for the conceptual structures of both teachers and 
students (see Figure 1), Stress=0.09 and RSQ=0.95 for teachers, Stress=0.09 and RSQ=0.96 for high-achieving 
students, with Stress=0.08 and RSQ=0.97 for low-achieving students. That is, the three-dimensional models are 
able to explain the data. 


Figure 2 
Three-dimensional Representation of the Conceptual Structure 
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Note: Dimension 1 denotes reactive and material conceptual attributes; Dimension 2 denotes relational and structural concep- 
tual attributes; Dimension 3 denotes fundamental and applied conceptual attributes, same below. 


In addition to the support of the above fitting data, the intuitive results of three groups (teachers, high- and 
low-achieving students) displayed by two-dimensional mapping images also reveal that the three-dimensional 
structure is a good reflection of the basic characteristics of chemistry (see Figures 3, 4, and 5). 

In Study 2, Dimension 1 reflects the shift from reactive to material structural attributes (from left to right, see 
Figures 3a, 3c, 4a, 4c, 5a, and 5c). The former includes concepts such as ion reaction, chemical equilibrium, and 
hydrolysis, etc., which are characterized by the dynamics of chemical reactions; the latter includes static concepts 
such as pure substance, mixture, and isotope, etc., which are related to the laws of substances. Dimension 2 reflects 
the shift from relational to structural conceptual attributes (from left to right, see Figure 3b, 4b, and 5b; from bottom 
to top, see Figure 3a, 4a, and 5a). The former includes concepts related to chemical structures such as solute and 
crystal; the latter includes the law of conservation of mass and periodic law of elements and some other concepts 
about the relationship between substances. Dimension 3 reflects the shift from fundamental to applied conceptual 
attributes (from top to bottom, see Figure 3b, 4b, and 5b; from bottom to top, see Figure 3c, 4c, and 5c). The former 
includes concepts related to daily or scientific research application, such as the greenhouse effect and acid rain; 
the latter includes concepts related to basic chemistry, such as molecules and ions. It can be seen visually in these 
figures that high-achieving students’ classification is more concentrated compared to low-achieving students. That 
is, high-achieving students are better able to establish inter-concept relationships based on their understanding 
of concepts when classifying them. 

In general, all three dimensions of spatial distribution for teachers and students reflected the characteristics 
of the chemistry discipline. However, the axis naming was based on the researcher's knowledge, experience, and 
cognitive ability, that is, experienced and cognitively competent researchers were able to summarize the charac- 
teristic commonalities of concepts based on their distribution in the coordinate axes. Therefore, the CA methods 
based on the three-dimensional coordinates were applied to make the classification clear and precise. 
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Figure 3 


ONE HUNDRED CORE CONCEPTS IN CHEMISTRY AND UPPER-SECONDARY SCHOOL 


TEAGHERS’ AND STUDENTS’ 


Two-Dimensional Projections of the Three-Dimensional Solution by Teachers. 
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Two-Dimensional Projections of the Three-Dimensional Solution by High-Achieving Students. 
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Two-Dimensional Projections of the Three-Dimensional Solution by Low-Achieving Students. 
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The dendrograms of CA results are shown in Figure 6. The measure ruler of the clustering dendrogram (see 
the top of Figure 6) indicates the adjustment distance of each concept in clustering. The concepts can be classi- 
fied into different categories by choosing different scale criterion. To show the results more visually, we clustered 
the concepts of teachers using “14” as the scale criterion and high- and low-achieving students into 10, 9, and 8 


clusters, respectively. 
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Figure 6 


The Dendrograms Obtained by Hierarchical Cluster Analysis of Different Groups. 
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Table 3 


The Classifications Obtained by Cluster Analysis 


Clusters of teachers 


1. Microcosmic 

2. Classifications of substances 
3. Chemical terms 

4. The law of conservation of mass 
5. Organic concepts 

6. Organic reaction 

7. Chemical amounts 


8. Chemical application 


Clusters of high-achieving students 


1. The law of conservation of mass 

2. Concepts related to the amount of substance 
3. Chemical application 

4. Organic reaction 

5. Organic concepts 

6. Concepts of redox reactions 

7. Electrochemistry and equilibrium 


8. Others 


(pp. 493-505) 


Clusters of low-achieving students 


1. The law of conservation of mass 

2. Concepts related to the amount of substance 
3. Organic reaction 

4. Chemical reaction 

5. Organic concepts 

6. Reduction 

7. Electrochemistry and equilibrium 


8. Others 


9. Energy 9. Chemical reaction 


10. Physical phenomenon 


Note: Concepts in bold indicate that they are unique to the group. 


The concepts within each cluster of teachers and students were further analyzed. It was found that the first 
and second clusters (i.e., microcosmic and classification of substances) of teachers contain many concepts within 
them, which could be further classified into sub-clusters. The first cluster contains five sub-clusters: 1) chemical 
characterization, 2) the periodic table of elements, 3) chemical bonds, 4) microcosmic structure, and 5) microcosmic 
concepts. The second cluster contains seven clusters: 1) concepts of solution, 2) stoichiometry, 3) classifications of 
substances, 4) chemical properties, 5) chemical reactions, 6) applications of chemical reactions, and 7) principles 
of chemical reactions and electrochemistry. The teachers’ conceptual structure begins with “classifications of sub- 
stances” and sequentially ties together with “chemical property’, “reaction”, and “application” to form the categories 
2. In general, the teachers’ classification was relatively complete and scientific. 

On the other hand, the clustering results of the high- and low- achieving students showed that the eighth 
clusters (i.e., others) contain many concepts within them, which could be further classified into nine sub-clusters and 
ten sub-clusters, respectively. For high-achieving students, the sub-clusters included 1) concepts of electrolytes, 2) 
basic concepts, 3) chemical bonds and valence, 4) concepts of microcosmic structure, 5) classifications and properties 
of substances, 6) ions, g) acid-base indicator, 7) chemical formula, and 8) rustiness. For low-achieving students, the 
sub-clusters included 1) microcosmic and classifications of substances, 2) chemical bonds, 3) electrolyte, 4) solution 
and energy, 5) physical change, 6) rustiness, 7) chemical calculation, 8) atomic group, 9) chemical application, 10) 
acid alkali salt, 11) substance properties, 12) average molecular weight, 13) structure, 14) chemical characterization. 

We found several common and different features between the two student groups’ conceptual structures. 
Regarding the common features, since some concepts, such as atomic clusters, are not found in chemistry text- 
books and exams, both groups did not classify them properly. That is, high-achieving students classified “atomic 
clusters” in category 4 (i.e., organic), but low-achieving students classified it in the category 8 (i.e., others); whereas 
the teachers classified it into the category 1 (i.e., microcosmic). Although both groups of students mixed many 
subcategories in category 8 (i.e., others), there were still some differences between the high- and low-achieving 
students. For example, the high-achieving students, in agreement with the teachers, classified “chemical application” 
separately, indicating that they had a better understanding of categorizing the concepts, such as acid rain, green 
chemistry, and the greenhouse effect while low-achieving students classified them as “others” Another example is 
that high-achieving students combined their middle and high chemical knowledge and grouped “acid, alkali, and 
salt” with “electrolytes” together, while the low-achieving students grouped them separately. 

In general, the conceptual structure differed significantly between high- and low-achieving students. Low- 
achieving students tended to classify concepts based on recent learning, and some concepts that had been learned 
too long ago were often classified separately and conceptualized confusingly. This indicates that low-achieving 
students’ conceptual structure in chemistry is incomplete, and the classification of concepts is confusing. 
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Discussion 


After determining the core concept in chemistry through personal interviews, material evaluation, and impor- 
tance rating questionnaire, this research investigated the conceptual structures of upper-secondary school teachers 
and students by using MDS and CA method, then further compared the differences in their conceptual structures of 
chemical concept. Two parts of the results were discussed as follows. 


The quality of core concepts 


The results of Study 1 showed that the data sources for the selection of core concepts are diverse and that the 
one hundred core concepts comprehensively cover the “basic structure” of the discipline of chemistry. 

The basic structure of a discipline, Brunner explains, refers to the basic concepts and principles of the discipline 
and their interrelationships. Only students who master this basic structure can achieve meaningful learning so that 
they can incorporate new material into their existing experience and establish a well-developed and systematic 
knowledge framework (Gulacar et al., 2022). From the one hundred core concepts in Study 1, the concepts of basic 
chemical units (e.g., acid and alkali salt), basic principles (e.g., the law of mass conservation), and changes in substance 
(e.g., various chemical reactions) were selected. These core concepts largely covered the“basic structure” of chemistry. 

Currently, most of the existing research on conceptual structure focuses on certain chemical topics and has a 
single method for concept selection. Mai and colleagues (2021) extracted twenty-four core concepts related to chemi- 
cal equilibrium using a relevance rating questionnaire. Gulacar and colleagues (2020), using word association test, 
asked students to list at least ten words they thought of when they read each of nine stimulus words related to core 
concepts commonly introduced in general chemistry. These studies reflect well the core concepts of certain chemical 
topics that researchers have focused on, but these core concepts are not yet representative of chemistry as a whole. 

Overall, the diverse and objective methods used in Study 1 ensured the scientific nature of the selected core 
concepts, and it is hoped that researchers will be able to use them as a general reference and fully implement them 
in future teaching processes. 


Conceptual structure 


The results of teachers’and students’ conceptual structures in Study 2 showed that three-dimensional solutions 
were appropriate for the conceptual structures. Each dimension was a good reflection of the basic characteristics of 
chemistry. 

Chemistry is the science of substances’ composition, structure, properties, and reactions and their laws and 
principles (K. Taber, 2013). An important purpose of chemistry is the formation of new substances through reactions, 
including separating a substance under certain conditions or forming new substances formed by multiple substances 
under certain conditions. According to the result of Study 2, Dimension 1 on the shift from reactive to material con- 
ceptual attributes well represents the essence of chemistry science. Dimension 2 reflects the shift from relational to 
structural conceptual attributes. According to some chemical educators, Chemistry is a subject that focuses on the 
relationship between the structures of various substances (Erduran & Kaya, 2019; Handtke & Bogeholz, 2022; K. Taber, 
2013). Clarifying these relations allows us to prepare the theoretical conditions for creating new substances. Dimen- 
sion 3 reflects the shift from fundamental to applied conceptual attributes. Chemistry is a science that falls halfway 
between basic and applied discipline (Doren & Duffy, 2016). The objective substances (e.g., molecules, atoms, etc.) 
and their structures and properties studied in chemistry often have a wide range of applications in real life, many of 
the research discoveries made in chemistry are closely related to the many aspects of daily living practice (Cooper 
et al., 2019). 

From a conceptual point of view, because the one hundred concepts selected in Study 1 cover the whole chemical 
discipline, the results of teachers’and students classification of these core concepts were more complex and scattered 
than in existing research that focused on a single topic in chemistry. For example, in the conceptual structure study 
of the atomic structure (Lin et al., 2022), the concepts related to atomic structure and electronic configuration were 
classified into one cluster, while the periodic table, periodic law and element properties were classified in another 
cluster. In the present study, the two clusters of concepts mentioned above appeared to be mixed in both teachers’ 
and students'classifications. This does not mean that the classification in Study 2 was unreasonable but may be related 
to chemistry textbooks and teachers’ teaching methods, which will be explained in the later part. 
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From the MDS results it can be seen visually that the classification of high-achieving students is more concen- 
trated than that of low-achieving students. That is, high-achieving students are better able to establish inter-concept 
relationships based on their understanding of the concepts when classifying them. 

The results could be due to the difference in meaningful learning ability between the two groups of students. 
Specifically, when learning a new concept, high-achieving students tend to make a conscious effort to find connections 
to the knowledge they learned previously. In the process, they can incorporate new concepts into their existing knowl- 
edge, thereby consolidating and extending their chemical knowledge framework, whereas the low-achieving students 
arbitrarily add new concepts to the existing knowledge framework. This rote learning style may encourage students to 
fail to apply the same tactic when the chemical problem is slightly different from the ones they have met before. These 
results were similar to the previous study. For example, Gulacar et al. (2022) divided students into high- and low-achieving 
students according to their previous performance in chemistry courses and also found that the conceptual structure of 
the high-achieving students was more organized, whereas the low-achieving students appeared disorganized, reflecting 
that they did not establish relationships between chemical concepts based on their understanding. 

As the differences in conceptual structure as determined by the MDS results are very subjective, it was necessary 
to further compare the differences in CA results between the two groups of students. For example, the CA results 
showed that there were differences between the two groups of students in category 6. The high-achieving students 
classified category 6 as “Concepts of redox reactions’, but the low-achieving students classified it as “Reduction” 

These differences between the two student groups may be due to the teachers’ teaching materials and the 
students’ learning order. In particular, the arrangement of concepts in textbooks is often the most important factor 
influencing students’ conceptual structure (Wolfer, 2000). In textbook compilation, relevant concepts are compiled 
in the same chapters, although these concepts do not necessarily belong to the same category due to their differ- 
ent nature. For example, students tend to put the “Colloid” and “Tyndall effect” together, perhaps because these two 
concepts are introduced in the same chapter of the textbook. As can be seen roughly from the clustering results in 
Study 2, this phenomenon is more common among low-achieving students. 

In addition, recently learned or reviewed concepts are more likely to be classified appropriately in the process 
of building conceptual structure. According to the recency effect, people tend to be familiar with the most recently 
presented information. This memory performance advantage may mean that low-achieving students are able to ap- 
propriately classify recently learned concepts, whereas for those learned early, they are more likely to classify them 
separately or incorrectly. This indirectly confirms previous findings that those who are able to understand more inter- 
conceptual relations are less influenced by external factors unrelated to the concepts themselves when understanding 
and classifying them because they have a more solid conceptual structure (Gulacar et al., 2022; Wilson, 1998). 


Conclusion and implications 


In Study 1, one hundred chemistry concepts were selected as core concepts. They were identified using per- 
sonal interviews, material evaluation, and the questionnaire method. Based on these core concepts, Study 2 further 
explored the conceptual structures of chemistry teachers and students in upper-secondary school using the MDS 
method, and the results showed that three-dimensional solutions were appropriate. In addition, this study also used 
CA methods to further explore the structural differences between high- and low-achieving students. The results 
showed that the conceptual structure of students with high-achieving students was more scientific than that of the 
low-achieving students. 

Based on the overall findings, there are several implications for future research. Firstly, our potential and most 
important aim of this research was to create a reference that can be used to analyze the conceptual structures of 
general chemistry of upper-secondary school students. Based on the viewpoint, the structure can be used as an 
assessment tool for teachers to predict why some students struggle with some concepts. Teachers can improve stu- 
dents’ learning by finding the parts of students’ cognition that do not have a reasonable conceptual structure and 
targeting their instruction. 

However, with the development of science and technology, many emerging substances are gradually synthe- 
sized by chemists, and many innovative techniques are gradually adopted. Students learn from diverse knowledge 
of chemical concepts through textbooks and the Internet, resulting in a change in the core concepts and conceptual 
structure in their minds. Therefore, teachers and students can perhaps use these core concepts and teachers’conceptual 
structure as a universal reference to adapt their teaching and learning objectives, while researchers can continue to 
incorporate new ideas in the future to create a more widely accepted standard concept and structure. 
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In addition, the diversity and variability of concepts and teachers’ and students’ conceptual structures can be 
seen as a prompt for future research. Specifically, future studies will be conducted to invite diverse populations to 
enable researchers to better understand expert thinking around chemical concepts, establish a more widely accepted 
preference for structure, and guide students to actively build a consolidated and refined conceptual structure. 
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Abstract. Physics homework often 

boils down to solving end-of-chapter 
quantitative problems. For targeting 
different learning goals of physics 
education, different types of homework 

are needed. The aim of this research was to 
compare the effectiveness of simulation- 
based, video-based, and paper-and-pencil 
homework in developing an understanding 
about Newton's laws and forming positive 
attitudes towards physics homework. 

150 first-year students from the Faculty 

of Chemical Engineering and Technology 
at the University of Zagreb (Croatia) were 
randomly assigned to one of the three 
above-mentioned homework approaches. 
After students had lectures and seminars 
on Newton's laws, they were administered 
a pre-test. In the next three weeks, the 
students completed three homework 
assignments on Newton's laws, after which 
they completed a post-test. For students 
from all three homework approaches a 
substantial improvement in conceptual 
understanding has been observed. 
Although the three approaches proved 

to be equally effective when it comes to 
developing understanding, the simulation- 
based approach was found to be superior 
when it comes to developing positive 
attitudes towards physics homework. If one 
controls for target knowledge, the modality 
of the homework assignment does not 
affect cognitive outcomes, but it does affect 
students’ attitudes towards homework. 
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Introduction 


One of the most important roles of the physics teacher is to design en- 
vironments which are very effective in activating, supporting, and sustaining 
those cognitive processes that are most relevant for students’ learning about 
a given physics topic. Thereby, the teacher is expected to prepare produc- 
tive learning environments not only for the classroom context but also for 
the homework context. In fact, learning physics is not limited to the physics 
classroom — much of physics learning may happen through the completion of 
homework. An important disadvantage of homework over classroom learning 
may be related to the fact that interaction with the teacher is largely missing. 
However, at the same time, a potential advantage of homework learning is 
that there are fewer time constraints and learners are in a better position to 
control the pace of their own learning. 

According to the model proposed by Trautwein et al. (2006), the effec- 
tiveness of homework learning is affected not only by the characteristics of 
the learning environment but also by students’ characteristics, homework 
motivation, parental behavior and homework behavior. Here the learning 
environment factor is mainly related to the quality and quantity of the home- 
work tasks, as well as to the teacher's feedback about students’ homework 
learning. How well the affordances provided by the learning environment are 
valorized, depends largely on the mere student, mainly on her/his cognitive 
ability and conscientiousness. However, for the relevant cognitive resources to 
be activated it is important that the student is motivated to do the homework. 
Concretely, she/he must perceive the homework activities as valuable and 
surmountable. This motivational aspect is important to start homework learn- 
ing, as well as to persist in completing homework activities. The mere learning 
process occurs through an interaction between the learner and the learning 
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environment. For the homework context, the main aspects of the learning environment are the characteristics of 
the homework tasks and parental behavior (e.g., monitoring homework completion, quality, and quantity of help). 
If this interaction between the learner, homework tasks (i.e., learning materials) and parents results in relevant and 
intensive cognitive processes, significant learning occurs. According to Sweller (1994), there are three mutually 
additive types of cognitive processes: relevant (i.e., processes that contribute to the construction of target knowl- 
edge), irrelevant (i.e., cognitive activities not related to constructing target knowledge) and intrinsic (i.e., related 
to the inherent complexity of the task). For most effective learning, a relevant cognitive load has to be fostered, 
irrelevant load minimized, and intrinsic load has to be adjusted (e.g., optimizing task structure) (Paas et al., 2003). 

It is very useful to realize that the only factor from Trautwein et al’s (2006) model which can be directly ma- 
nipulated by the teacher is the learning environment. Concretely, teachers may significantly influence the quality of 
homework learning experiences by carefully choosing homework tasks. In practice, conventional physics homework 
boils down to answering a few end-of-chapter questions that assess factual or conceptual knowledge, followed by 
the requirement to solve some quantitative problems. Thereby, often these questions and problems are chosenina 
random fashion, instead of being carefully selected to allow for an optimal upgrade of competencies developed in 
the science classroom (Me&i¢ et al., 2022). In addition, the typically assigned end-of-chapter questions and quanti- 
tative problems do not optimally reflect the spectrum of learning goals we tend to achieve in modern-day science 
education, such as goals related to science inquiry skills or goals related to developing positive attitudes towards 
science. Here attitudes towards science are defined as “feelings, beliefs and values held about an object that may be 
the enterprise of science, school science, the impact of science on society or scientists themselves” (Osborne, 2003, p. 
1053). Some of the components of the attitude towards science construct which are mentioned in earlier research 
are: motivation towards science, self-esteem in science, perception of achievement in science and perception of 
science classroom environment (Osborne et al., 2003). It is useful to distinguish between attitudes towards science 
and attitudes towards school science. Attitudes towards school science are the “product of students’ experience of 
school science” (Osborne, 2003, p. 1055), and it is exactly this construct which largely influences students’ career 
choices and their readiness for life-long learning of science. Considering that the number of students who choose 
careers in physics is continually decreasing and the societal needs for physicists are increasing (Kennedy et al., 
2014; Oon & Subamaniam, 2011; Pronovost et al., 2016; Roach & Sauermann, 2017), it is very important to find ef- 
fective ways for developing positive attitudes towards school physics in the students. When it comes to homework 
assignments, some alternatives to typical end-of-chapter tasks have to be found. 

Unlike end-of-chapter questions and problems, learning-by-inquiry activities are very well suited to target 
a wide spectrum of different learning goals, including the goal of developing a positive attitude towards school 
physics (Simsek & Kabapinar, 2010). However, learning-by-inquiry is known to be relatively time-consuming, which 
is one of the reasons why it is being avoided by some teachers, especially if they have only a small number of teach- 
ing hours at their disposal. Consequently, it may be useful to prepare learning-by-inquiry activities for the context 
of physics homework. Considering the fact that students generally love to learn with contemporary technologies 
(Li, 2007; Mesi¢ et al., 2022), a practical solution would be to situate inquiry-learning activities within the contexts 
of simulations or digital videos. A simulation can be defined as an “imitation of a real-world process or system over 
time” (Banks et al., 2010, p. 3). A particularly prominent feature of physics simulations is that it is very easy to change 
certain physical parameters and observe corresponding changes of the visually represented real-world processes 
or systems over time. In other words, the learner easily explores cause-and-effect relationships, which facilitates 
the development of functional mental models about physics phenomena (Dervic¢ et al., 2018). On some occasions, 
virtual experiments are believed to be an even better option compared to experiments with real equipment (De 
Jong et al., 2013). For example, this is the case when real setups are associated with considerable health risks or 
when manual data collection would be very time consuming. In addition, it is asserted that simulations provide 
implicit scaffolding (Moore et al., 2013; Roll et al., 2018), which is particularly important within the homework 
context because direct interaction with the teacher is missing. Concretely, well-designed simulations allow the 
learner to use the virtual equipment only in ways that support the achievement of learning goals — the number of 
degrees of freedom for using the virtual equipment is typically lower than for using real equipment. Consequently, 
irrelevant cognitive load is lower for virtual experiments, which means more working memory capacity for running 
relevant cognitive processes. However, which cognitive processes will be run largely depends on the tasks the 
students are expected to implement within the context of a simulation. The tasks should be designed to activate 
cognitive processes which are recognized as very important for learning a given physics topic. For example, within 
the context of Newton's first law, thinking about gradually decreasing friction is one of the key cognitive processes. 
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Generally, for developing conceptual understanding “predict-observe-explain” activities proved to be useful in 
earlier research (Karamustafaoglu & Mamlok-Naaman, 2015). 

To make sure that all students engage in fruitful cognitive processes, explicit scaffolding may be provided 
through carefully designed worksheets (Adams et al., 2015). These worksheets can include prompts that facilitate 
the planning and implementation of inquiry activities, as well as promote monitoring and self-evaluation of the 
learning process. 

Another technologically rich approach to physics homework would be to situate homework tasks within the 
context of videos (Laws et al., 2015). Videos may be used for purely informative purposes or for organizing learning 
by inquiry. The inquiry-variant of video-based homework may require the learner to analyze the video qualitatively 
or quantitatively to be in a position to answer certain questions that target key cognitive processes related to the 
given physics topic. Unlike interactive simulations, videos do not offer the opportunity for hands-on exploration 
of cause-and-effect relationships. However, videos may provide the students with the opportunity to observe and 
analyze physical phenomena to come to certain conclusions about relationships between physical quantities. 
On the other hand, there are also approaches in which the learner is expected to perform measurements from 
the videos on their own with the aim of setting up models of physical phenomena (e.g., in digital video analysis) 
(Brown & Cox, 2009). 

Whatever homework approach is used for learning about Newton's laws, it is important that the homework 
tasks provoke cognitive processes that are at the core of Newtonian mechanics. In mechanics, it is important for 
the students to learn the difference between acceleration and velocity, as well as the difference between the force 
and effects of force (Aviani et al., 2015; Erceg & Aviani, 2014). Force must be perceived as a measure of interaction 
between some physical entities. Thereby, students should realize that the forces of action and reaction must be of 
the same nature and that they never act on the same body (Knight, 2015). They also must realize that a change of 
the velocity vector is always associated with a non-zero resultant force and that objects may sustain their motion, 
even if there are no forces acting on them. Here it is also important that students overcome the misconception that 
static objects like tables cannot exert forces (Clement, 1993) and that they develop the ability to correctly identify 
forces, including the force of friction which is often not regarded as the force by students (Kizilcik et al., 2021). 


Research Aim 


There are not many studies about the relative effectiveness of simulation-based, video-based, and paper-and- 
pencil homework. In the study by MeSi¢ et al. (2022), simulation-based homework proved to be substantially more 
effective than paper-and-pencil homework when it comes to developing a conceptual understanding about gas 
laws in the upper-secondary school context. In a similar study by Simi¢ et al. (2022), simulation-based homework 
proved to be equally effective as video-based and paper-and-pencil homework when it comes to developing 
conceptual understanding about energy and work, but significantly more effective than any of these two ap- 
proaches when it comes to developing positive attitudes towards physics homework in upper-secondary school 
students. Therefore, one can say that the findings from earlier research on simulation-based physics homework are 
mixed and mostly situated within the upper-secondary school context. In addition, a significant limitation of the 
above-mentioned two studies is related to the fact that for the paper-and-pencil group, the homework questions 
were not carefully selected to target as similar as possible conceptual knowledge compared to questions from the 
simulation-based and video-based groups. 

The aim of this research was to compare the effectiveness of simulation-based (SB), video-based (VB) and 
paper-and-pencil (PP) physics homework when it comes to developing an understanding about Newton’s laws 
and forming positive attitudes towards physics homework in university students. Concretely, the research ques- 
tions were as follows: 

1. How effective are the SB, VB and PP approaches, when it comes to developing a conceptual understand- 
ing about Newton's laws in university students? 

2. How effective are the SB, VB and PP approaches, when it comes to forming a positive attitude toward 
physics homework in university students? 

Considering that students are generally fond of contemporary technologies (Li, 2007), it has been expected 
that simulation-based and video-based approaches are superior to the paper-and-pencil approach when it comes 
to developing positive attitudes towards physics homework. When it comes to the development of conceptual 
understanding, the potential advantage of the simulations is mainly related to the interactivity feature (Kalyuga, 
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2008), whereas a potential advantage of the video-based approach is that students receive more guidance (e.g., 
technical aspects of the given procedures were performed by the teacher) which potentially leaves the students 
more time for thinking. Finally, a potential advantage of the paper-and-pencil approach is related to a more familiar 
format of the homework tasks and simpler task formulations (i.e., less irrelevant load, and less intrinsic cognitive 
load). However, for the paper-and-pencil approach, there is a lack of dynamic visualizations and interactivity, which 
may be an important disadvantage of this format, leading to less effective homework learning. 

The significance of this research is related to the fact that it describes in detail two alternative approaches to 
physics homework (i.e., simulation-based, and video-based physics homework) and their effects on some aspects 
of physics learning. In the current teaching practice, physics homework often fails to target a wide spectrum of 
learning goals (cognitive as well as affective ones) that are important for physics education. Concretely, students 
are typically asked to solve a few end-of-chapter, quantitative problems which potentially contribute to develop- 
ing students’ problem-solving skills, but fails to target some other, important learning goals, such as developing 
science inquiry skills or developing a positive attitude toward physics. Consequently, there is a need for research- 
based, alternative approaches to physics homework which could help the teachers to target a wider spectrum of 
learning goals. For example, by identifying homework approaches that are perceived positively by the students, 
we can expect more students to become ready for lifelong learning of physics and to choose a career in physics. 
On the other hand, having more young people who choose their career in sciences is very important for meet- 
ing the demands of technology-based societies (Maltese & Thai, 2011). In addition, comparing video-based and 
simulation-based homework might be a significant contribution to physics education literature, because some- 
times the students do not have at their disposal a certain simulation in their home environment, but the teacher 
can screen-record the simulation and ask them to analyze the corresponding video at home. Therefore, it might 
be useful to compare the effectiveness of simulation-based and corresponding video-based homework, across 
various physics topics and student populations. 


Research Methodology 
Research Design 


A pre-test - post-test true experimental study with three comparison groups has been conducted (Ary et al., 
2009). The research design is described in Table 1. 


Table 1 
Research Design 


Week 3 of semester 


2 hours of lectures and 
2 hours of seminars on 
Newton’s laws (same 
for all students) 


Week 4 of semester 


Random assignment 
of the students to the 
three homework ap- 
proaches 


Week 5 of semester 


Conceptual pre-test on 
Newton's laws 


Homework on New- 
ton’s first law 


Same lectures and 
seminars on other 
topics 


Week 6 of semester 


Homework on New- 
ton’s second law 


Same lectures and 
seminars on other 
topics 


Week 7 of semester 


Homework on New- 
ton’s third law 


Same lectures and 
seminars on other 
topics 


Week 8 of semester 


Conceptual post-test 
on Newton's laws 


Survey on attitudes 
towards school physics 


Same lectures and 
seminars on other 
topics 


The conceptual pre-test on Newton's laws has been conducted after all the students had the same lectures and 


seminar sessions on Newton's laws. This allowed us to associate the pre-test-post-test conceptual gain solely with 
the effect of homework learning. Considering that random assignment has been conducted and that the group 
sizes were relatively large, one can be pretty confident that the three student groups (SB, VB, PP) were comparable 
across all latent traits at the moment of entering the homework interventions (Ary et al., 2009). After week four 
the only systematic effort related to learning about Newton's laws was the physics homework. The homework 
was different for the SB, VB and PP groups. Consequently, potential between-group differences on the conceptual 
post-test and attitude survey may be validly related to differences between the three homework approaches. 
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The pre-test and post-test have been administered within regular seminar sessions, while the attitude survey 
has been administered as a Google Form and filled out by the students outside of university classes. For purposes 
of easier and more valid pre-test - post-test comparisons, the same conceptual test has been used as a pre-test and 
post-test. For all students, the lectures were given by the same teacher. All research activities from Table 1 were 
conducted between October 24" 2022 and December 4" 2022. 


Participants and Curriculum 


The target population consisted of first-year engineering students in introductory, two-semester physics 
courses. In this research, the student sample included 150 first-year students (mostly 19-year-olds) from the Faculty 
of Chemical Engineering and Technology at the University of Zagreb, Croatia. All participants gave their informed 
consent to participate in the study after being made aware of its purpose. They were assured that the principles 
of anonymity and confidentiality would be adhered to in the study. 

All 150 students who attended the Physics | seminar sessions in the fifth (pre-test) and eighth week (post-test) 
of the semester were included in the sample for drawing conclusions about the cognitive effects of the SB, VB and 
PP homework approaches. Out of these 150 students, 122 were female and 28 were male. On the other hand, the 
attitude survey has been filled out by 137 students (114 female, 23 male). The students were randomly assigned to 
one of the three homework approaches: simulation-based (n,, = 51), video-based (n,, = 48) and paper-and-pencil 
lee =51). 

In their first year of studies, students from the given sample enroll in the Physics | (fall semester) and Physics II 
(spring semester) courses. These courses include two teaching hours of lectures and two teaching hours of seminar 
sessions per week. In the Physics | course two hours of lectures and two hours of seminar sessions are devoted 
to learning about Newton’s laws. Lectures mostly focus on facts and mathematical procedures, while seminar 
sessions focus on solving numerical problems. Concretely, in the seminar sessions most of the time, the teaching 
assistant is modelling the problem-solving process to a class consisting of 30-40 students. Generally, the Physics | 
and Physics II curriculum include all the physics topics that are typically presented in widely known introductory 
physics textbooks, such as Physics for Scientists and Engineers by Serway and Jewett (2013). 


Description of Homework Approaches 


Here simulation-based homework boils down to learning physics by interacting with a simulation and complet- 
ing the corresponding worksheet prepared by the teacher. For the video-based homework, the only difference to 
simulation-based homework is related to the fact that students do not interact with the simulation on their own, 
but they learn by analyzing the screen-recorded version of the simulation that had been originally manipulated 
by the teacher. Again, nearly the same worksheet is completed as in the simulation-based group. Finally, for the 
paper-and-pencil group, the tasks resembled as much as possible the tasks from the simulation-based and video- 
based treatments. For example, the students were asked to perform a thought experiment instead of conduct- 
ing/observing a virtual experiment, or they were provided with the same experimental data (as was collected by 
students in SB) and asked to analyze it. 

It is important to note that the paper-and-pencil homework approach from this research cannot be considered 
as conventional physics homework, because its tasks are not typical end-of-chapter questions and problems. How- 
ever, unlike VB and SB homework, PP homework did not allow for the collection of measurements and observation 
of processes. In all three approaches, the activities mainly included identifying patterns and applying knowledge 
for predicting the outcome of certain processes. The VB and SB worksheets often included tasks related to the 
exploration of relationships, as well as predict-observe-explain activities. For purposes of providing a vivid insight 
into the similarities and differences between the three homework approaches, a complete, translated version of 
the materials for homework on Newton's second law is given at the following link: http://fizika.pmf.unsa.ba/wp- 
content/uploads/2023/02/Newton_2nd_law_worksheets.pdf 

Significant efforts have been invested in targeting the same key conceptual ideas in all three homework ap- 
proaches. To that end, paper-and-pencil homework questions were developed, only after the worksheets for SB 
and VB approaches had been completed. The prominent Newtonian ideas that were targeted through homework 
activities in all three groups are described in Table 2. 
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Table 2 


Prominent Newtonian Ideas Targeted through Homework Activities 


Newton’s first law 


Velocity of an object does not change if and only 
if the resultant force acting on the object is zero. 


An object may rest even if we apply a force on 
it (e.g., if there is a friction force of the same 
magnitude but opposite direction, acting on it). 


Any object tends to maintain its velocity (magni- 
tude as well as its direction); mass is the measure 
of inertia. 


Newton’s second law 


Non-zero net force (i.e., unbalanced forces) acting 
on an object changes the velocity of that object. 


Type of object's motion depends on the sum of 
all forces acting on that object; non-living objects 
also may exert a force and static friction forces 
can vary from zero to a maximum value. 


If net force and velocity vector have the same 
direction, at that moment velocity magnitude is 
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Newton’s third law 


When two objects interact, a pair of forces oc- 
curs; these forces are of the same nature and 
magnitude but act on different bodies and point 
in mutually opposite directions. 


Effects of force application do not depend only on 
the magnitude of a force but also on the mass on 
which the force is acting (e.g., in the collision of a 
truck and car, forces of the same magnitude act 
on both, but the deformation of the car is larger). 


increasing; if they are mutually opposite, mag- 
nitude decreases. 


Three Google classrooms have been created for the SB, VB and PP groups of students. Each Saturday morning, 
starting from the fifth week of the semester, the homework assignments were uploaded to the Google classrooms. 
The SB and VB students were provided with links to the simulations/videos, respectively, and they were also 
provided with corresponding worksheets. On the other hand, the PP students were only provided with a sheet 
containing questions and problems. All students have been asked to complete the homework before Sunday and 
upload the scans of their homework solutions to the Google classrooms. To motivate the students to invest more 
effort into completing homework, the 10% of students with the highest homework scores were promised to get 
some extra credits which could affect their final grade in Physics | course. On Sundays, the teacher uploaded the 
official homework solutions to the Google classrooms and asked the students to compare these official solutions to 
their own solutions, i.e., she encouraged them to perform self-reflection and to report time invested in homework 
completion. The teacher also scored all the submitted homework solutions; each submitted homework solution 
was assigned with a percentage score, based on the correctness of students’ reasoning and final solutions. 


Instruments 


For the research aim to be fulfilled, evidence on students’ conceptual understanding of Newton’s laws, as well 
as on their attitudes towards physics homework, had to be collected. According to Goldwater and Schalk (2016), 
conceptual understanding is relational knowledge about the core concepts in a domain and their interrelations. 
For the domain of mechanics, the core concepts are displacement, velocity, acceleration, mass and force. Some 
of the cognitive processes that may reflect conceptual understanding are: interpreting, comparing, identifying, 
predicting and explaining (Anderson & Krathwohl, 2001). 

Based on these theoretical considerations about the construct of conceptual understanding, as well as on the 
list of prominent Newtonian ideas from Table 2, 20 multiple-choice items with a single correct answer and three 
distracters were selected from existing literature or newly created. Concretely, 11 out of the 20 items were adapted 
(e.g., compared to original items only the number of distracters may have been changed) from earlier published 
literature, such as: Force Concept Inventory (Hestenes et al., 1992), Mechanics Baseline Test (Hestenes & Wells, 
1992), Force Body Diagram Test (Aviani et al., 2015), TIPERS (Hieggelke et al., 2015) and College Physics by Knight 
et al. (2015). The same 20 items were used for the pre-test and for the post-test. Ideas targeted by the individual 
items are briefly described in Table 3. 
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Table 3 


Brief Description of Ideas Targeted by the Conceptual Test 
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ISSN 2538-7138 /oniines 


Item 1 
F,, <0 > v=const, context of 
smooth surface 


Source: Original 


Item 2 
v=const#0 > balanced forces 


Source: FCI (item 25) 


Item 3 

unbalanced forces > v#const; 
F 4. and v in same direction >v 
increases 

Source: FCI (item 26) 


Item 4 

unbalanced forces > véconst; 
F 4.and v opposite >v de- 
creases 

Source: FCI (item 27) 


Item 5 

v=0 > balanced forces 
non-living objects may exert a 
force 

Source: FBDT (item 1) 


Item 6 
v=const#0 > balanced forces; 
identifying forces 


Source: FBDT (item 3) 


Item 7 


v increasing > unbalanced forces; 


Fin the same direction as v 


nei 


Source: FBDT (item 4) 


Item 8 
Objects 1 and 2 interact, m,>m, 
>F,,=F,,, context of pushing off 


12° 21) 


Source: FCI (Item 28) 


Item 9 
Objects 1 and 2 interact, m,>m, 
DF =F, a,<a, 


Source: Original 


Item 10 

An object resting in a moving 
vehicle tends to maintain v, when a 
vehicle stops 

Source: Original 


Item 11 

Objects 1 and 2 interact, m,>m, 
> F,,.=F,,, context of collision 
Source: Original 


Item 12 
balanced forces > v=const 


Source: Original 


Item 13 

A force F with a direction opposite 
to v tends to decrease v 

Source: Original 


Item 14 
Objects 1 and 2 interact, m,>m, 
DF =F ,,7a,<a, 


Source: Original 


Item 15 
F__=0 > v=const, context of 


net 


Space 


Source: Original 


Item 16 
Linear increase of v > F=const#0 


Source: MBT (item 3) 


Item 17 
v=const#0 > balanced forces; 
context of elevator 


Source: FCI (item 17) 


Item 18 

v=0 while F, and F,, act on the 
object >F =F, not an action- 
reaction pair 

Source: Original 


Item 19 

Larger v generally does not imply 
larger F acting on the body 
Source: Knight et al. (2015) 


Item 20 

Objects 1 and 2 interact, m,>m, 
>F,,=F,,, context of vector 
representation 

Source: TIPERS (B3-QRT96) 


Since eleven items were adapted from widely accepted literature, and the other nine items covered mostly the 
same Newtonian ideas but situated in different contexts, it can be concluded that the given test may be used for 
drawing valid conclusions about university students’ conceptual understanding of Newton's laws. Next, reliability 
analyses for the conceptual test have been conducted. For item 18 the item-total correlation proved to be negative 
and consequently, this item has been removed from the scale. The Cronbach’s alpha of the 19-item scale amounted 
to .71 for the pre-test and .79 for the post-test. Both values indicate reliable test scores (Cohen et al., 2007). 

For measuring students’ attitudes towards physics homework, a shortened version of the attitude survey de- 
veloped by Me&i¢ et al. (2015) has been used; the same instrument has also been used in the homework research 
by Simi¢ et al. (2022). The attitude survey initially included six statements for which students were expected to 
express their level of agreement on a 4-point Likert scale (Table 4). 


Table 4 


Attitude toward Physics Homework Items 


Item 1 


The homework on Newton's laws made me 
interested to learn more about this topic. 


Item 2 


| feel that the homework on Newton's laws 
helped me to considerably improve my knowl- 
edge about Newton's laws. 


Item 3 


Although | invested efforts into the homework 
on Newton’s laws, it was difficult for me to 
complete it. 


Item 4 


While doing the homework on Newton’s laws, | 
have been learning physics with understanding. 


Item 5 


Learning about Newton’s laws boils down to 
memorizing facts and equations, and math- 


Item 6 


ematical solving of these equations. 


We should more often do homework like the 
one on Newton’s laws. 


AS 
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Eventually, reliability analyses showed that Item 5 from the attitude survey was characterized by a negative 
item-total correlation and consequently it has been excluded from the attitude towards physics homework scale. 
The reliability of the final attitude scale was found to be .71 which is regarded as reliable, particularly for such a short 
scale as the one used in this research (Cohen et al., 2007). In addition, the students were also asked the following 
open-ended question: “What is your general impression about the Newton's laws homework?" 


Data Analysis 


For students who wrote the pre-test as well as the post-test, the answers of each student on each item were 
entered into SPSS. Thereby, the response option “a” has been coded as“1",“b"as“2",“c"as“3",and“d” as “4” Next, the 
answers have been recoded - correct answers were coded as “1” and incorrect as “0”. After the reliability analysis 
had been conducted, composite scores on the pre-test and post-test were calculated by simple summing of scores 
across individual items. Initially, the plan was to use the typical parametric tests for checking: 1) whether the pre- 
test-post-test gain was significant (paired t-test) 2) whether between-group differences on post-test were significant, 
after controlling for pre-test differences (ANCOVA). The most important assumptions for the classic paired t-test 
were met: Shapiro-Wilk test and Kolmogorov-Smirnov tests of normality were non-significant for pre-test-post-test 
differences within the three groups and there were no extreme outliers in the differences. However, for ANCOVA the 
statistical assumption of normal distribution of test scores within the groups was not met. Consequently, instead 
of classic ANCOVA, the WRS2 R package has been used to conduct a robust version of ANCOVA (Field, 2018; Mair & 
Wilcox, 2020). Considering that robust ANCOVA can be conducted for two groups at one time only, for the purpose 
of conducting ANCOVA two separate databases have been created: one included SB and VB data, and the other 
included SB and PP data. This made it possible to compare SB versus VB, and SB versus PP. 

When it comes to the attitude towards the homework part of the study, students’ Google form answers were 
exported to an SPSS data file. For the attitude towards homework items, the following answer coding has been 
used: 1-Fully disagree, 2 - Mostly disagree, 3 - Mostly agree, 4- Fully agree. Initially, items 3 and 5 were negatively 
keyed and consequently, they had to be reverse coded. After item 5 was removed due to negative item-total cor- 
relation, a composite score was calculated by summing the five positively keyed items. Because the assumptions 
of the classic ANOVA were not met (e.g., the variance in SB was much smaller than in the other two groups), a 
robust variant of one-way independent ANOVA with post-hoc tests, as described by Field (2018) has been used 
to analyze the significance of between-group differences on the attitude scale. In addition, chi-square tests have 
been conducted for checking for between-group differences on individual Likert-type items. All analyses have 
been performed in IBM SPSS 26 and R software, version 4.2.2. 


Research Results 


Descriptive results from the conceptual pre-test and post-test are reported in Table 5. The test scale ranged 
from 0 to 19 points. 


Table 5 
Mean Scores (M), Standard Deviations (SD) and Standard Errors (SE) at Pre-Test and Post-Test 


Simulation-based approach Video-based approach Paper-and-pencil approach 
(n,,=51) (n,,=48) (n,,=51) 
Pre-test M= 8.49; SD = 3.38; SE = 0.47 M= 7.77, SD = 3.33; SE = 0.48 M= 7.75; SD = 3.24; SE=0.45 
Post-test M= 14.29; SD = 3.75; SE = 0.52 M= 14.12; SD = 3.52; SE = 0.51 M= 14.25; SD = 3.19; SE = 0.45 


The pre-test-post-test difference for the VB approach proved to be statistically significant, t(47) =-11.88, p< .001, 
and represented an effect of d= 1.91. This is a large effect. In fact, the percentage of correct answers increased from 
40.9% to 74.3%. The pre-test-post-test differences for SB and PP approaches were also statistically significant; for 
SB t(50) = -11.98, p < .001 and for PP t(50) = -12.09, p < .001. For the SB approach, there was an effect of d= 1.71, 
and for the PP approach an effect of d = 2.00. In the SB group, the percentage of correct answers increased from 
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44.7% to 75.2%, and in the PP group from 40.8% to 75.0%. Next, the post-test differences between SB and PP were 
analyzed, while the pre-test scores were controlled (Table 6). 


Table 6 
Results of Robust ANCOVA for SB versus PP Comparison 


n,, Np M57 Mp, Lower Cl Upper Cl Statistic p value 
Pre-test = 4 20 18 -1.25 -5.30 2.80 -0.87 Al 
Pre-test = 5 22 19 -0.89 -4.57 2.78 -0.69 ot 
Pre-test = 7 31 28 -0.94 4.13 2.23 -0.84 A2 
Pre-test = 9 20 27 0.26 -2.85 3.38 0.24 19 
Pre-test = 12 12 12 1.25 -1.54 4.04 1.26 25 


The analysis of post-test differences between SB and VB, while controlling for pre-test scores, is presented in 
Table 7. 


Table 7 
Results of Robust ANCOVA for SB versus VB Comparison 


n,, ny M4” May Lower Cl Upper Cl Statistic p value 
Pre-test = 4 20 17 -0.04 -5.37 5.30 -0.02 99 
Pre-test = 5 22 24 -0.61 -5.02 3.80 -0.39 69 
Pre-test = 6 27 25 -1.24 -4.96 2.48 -0.95 35 
Pre-test = 9 20 24 0.56 -2.53 3.66 0.52 60 
Pre-test = 11 18 15 0.83 -1.19 2.85 1.17 .28 


Next, the between-group differences on the attitude scale were analyzed. The mean attitude scores in the 
SB, VB and PP groups were 17.89, 16.25 and 15.16, respectively (the theoretical maximum of the scale was 20). A 
bootstrapped version of robust ANOVA showed that there were statistically significant between-group differences 
on the attitude towards physics homework scale, F, = 19.34, p < .001. The corresponding effect size amounted to 
.51 which can be considered as large (Mair & Wilcox, 2020). In addition, the post-hoc tests showed that the SB ap- 
proach was superior to VB (difference in trimmed means = 1.30, p = .02), as well as to PP (difference in trimmed 
means = 2.47, p < .001). The VB approach proved to be superior to PP (difference in trimmed means = 1.18, p = .04). 
It should be noted that in all these comparisons the p values were adjusted for multiple testing (Mair & Wilcox, 
2020). Finally, chi-square tests were conducted to explore between-group differences on individual Likert-type 
items. Statistically significant between-group differences were observed for item 6 (x?(6) = 28.03, p < .001) and item 
4 (y?(6) = 24.94, p < .001) from the attitude survey. Figure 1 shows how students from different groups answered 
to these two items. 


514 


URS https://doi.org/10.33225/jbse/23.22.506 


Journal of Baltic Science Education, Vol. 22, No. 3, 2023 


ISSN 1648-3898 | /erinty SIMULATION-BASED AND VIDEQ-BASED APPROAGHES TO DIVERSIFYING PHYSICS 

HOMEWORK 

ISSN 2538-7138 /oniine (pp. 506-519) 
Figure 1 


Attitude Items with the Largest Between-Group Differences 


80 80 
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Percent 
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Mostly agree 
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ZA is 
Fully disag! 


B__ 


Mostly disagree 


Fully agree 


Mostly disagree 


Attitude 4 (learned with understanding) Attitude 6 (should more often have homework like this) 


When it comes to the open-ended question asking about the general impression, most students provided brief 
answers indicating a positive impression (e.g., “was OK’, “very good”). It may be most informative to quote selected 
answers, from different groups. For example, one student from the PP approach stated:"I liked the homework questions 
because they motivated me to think and for each question, we could answer something, even something wrong. Later, we 
could see where our reasoning, i.e., our perception of the situation went wrong.” One student from the SB group made the 
following statement:“/t made me explore Newton's laws in more detail and find out what they are all about. !am absolutely 
certain that now | better understand this topic.” Finally, one student from the VB group asserted the following: “More in- 
teresting than mere quantitative problem solving in the classroom and, | repeat it once again, better understanding through 
visualized situations.” However, some other students, mainly from the VB group, expressed also negative experiences, 
such as: “Some problem statements were not clearly specified”. Students from the VB group reported that they, on aver- 
age, invested 90 minutes in completing the homework, while in the SB group and PP group, the average homework 
time amounted to 80 minutes and 52 minutes, respectively. 


Discussion 


All three homework approaches resulted in statistically significant and large improvements in conceptual un- 
derstanding of Newton's laws. This finding may be explained by Trautwein et al‘s (2006) model of homework learning. 
Concretely, students solved the homework assignments after they had lectures and seminar sessions on Newton's laws. 
This means that they entered the homework-solving process at least with some basic foreknowledge. They were also 
extrinsically motivated to engage in the homework-solving process because extra credits were promised to students 
who achieve top homework scores (Cheng et al., 2004). In the VB and SB groups, the visually rich and/or interactive 
learning environment provided an additional source of (intrinsic) motivation. Even more important is that the home- 
work tasks were designed to reflect key Newtonian ideas, such as the ones presented in Table 2. Students’ answers 
from the attitude survey show that most students perceived the homework tasks as challenging and interesting, which 
motivated them to invest significant time and cognitive resources into the completion of homework. Thereby, much of 
the cognitive load was related to implementing relevant cognitive processes (e.g., more than 80% of students from all 
groups agreed with the statement that they learned about Newton's laws with understanding), which is key to effective 
learning (Sweller, 1994). Some students also lauded the fact that they could learn from the teacher's feedback, which 
is in line with earlier findings on the importance of feedback about homework (Tas et al., 2016; Xu, 2016). 

All three homework approaches proved to be equally effective when it comes to developing a conceptual under- 
standing about Newton's laws. The potential for generating relevant cognitive load was probably somewhat higher 
in SB and VB than in PP, because SB and VB allowed for interactivity and/or learning with multiple representations 
(Ainsworth, 2006; Kalyuga, 2008). However, in SB and VB the intrinsic and irrelevant load were higher too (e.g., due 
to more complex homework instructions and due to learning how to do homework with simulations or videos). For 
some SB and VB students, this certainly resulted in cognitive overload. Data from Table 6 suggest that this cognitive 
overload mostly occurred in SB students with low level of foreknowledge. The potential of the SB approach seems to 
be better valorized by students with a higher level of foreknowledge. 
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When it comes to the development of conceptual understanding, the results from this research are in line with 
the work by Simic et al. (2022) who found that simulation-based, video-based, and paper-and-pencil homework were 
equally effective in developing conceptual understanding about energy and work. Consequently, the findings by 
Simic et al. (2022) related to the equal effectiveness of simulation-based and corresponding video-based homework 
may be extended to university students and some other physics topics, such as Newton's laws. On the other hand, the 
results from this research seem to be not in line with the results by MeSic¢ et al. (2022) who found that the simulation- 
based homework was significantly more effective in developing students’ conceptual understanding about gas 
laws compared to paper-and-pencil homework. However, in that study, the paper-and-pencil homework included 
typical end-of-chapter problems, i.e., the key conceptual ideas were not carefully controlled across the homework ap- 
proaches. In addition, unlike the Newton's laws simulations, the gas laws simulation visualized abstract processes on 
the level of micro-particles which was more difficult to translate into paper-and-pencil homework. The effectiveness 
of simulation-based homework, when it comes to producing substantial learning gains, is in line with the findings by 
Adams et al. (2015). They showed that simulation-based homework which includes carefully scaffolded worksheets 
may help the students to achieve large learning gains, comparable to learning gains related to simulation use in the 
classroom setting. According to Wieman et al. (2010), a specific strength of simulation-based homework is that it al- 
lows the exploration of phenomena related to, but different from the ones observed in the class (e.g., high altitude 
breathlessness). The opportunity to analyze a wide spectrum of different, vivid phenomena is also a characteristic of 
the video-based physics homework. In the study by Laws et al. (2015), out-of-class learning with interactive videos 
resulted in a significantly increased conceptual understanding of Newton's laws. Similarly to videos from this research, 
the videos from the study by Laws et al. (2015) required the students to carefully observe physics phenomena and 
make predictions. This is consistent with de Araujo and Otten’s (2017) idea of improving the quality of video-based 
homework through the inclusion of interactive elements (e.g., applets, quizzes). They also point out the importance 
of systematically associating homework activities with specific learning goals. 

The simulation-based approach proved to be superior to the other two approaches when it comes to develop- 
ing a positive attitude toward physics homework. This result is in line with earlier research (Mesi¢ et al., 2022; Ronen & 
Eliahu, 1999; Simi¢ et al., 2022). In fact, most students like to learn with contemporary technologies (Li, 2007). Concretely, 
students praise simulations for providing them with vivid feedback about their own learning, which is particularly 
important in the homework context, where the teacher is absent (Ronen & Eliahu, 1999). Generally, students like to 
have an active role and to control the pace of their learning (Fouts & Myers, 1992; Kay & Edwards, 2012; Maltese & Thai, 
2011). Unlike in the SB approach, in the VB approach the students did not have the opportunity to interact on their 
own with the simulation. They were asked to stop the video at certain instants to answer some questions which means 
that their level of control of learning was lower than for the students in the SB context. This may explain the superiority 
of SB to VB when it comes to developing positive attitudes towards physics homework. 


Limitations 


Since the same conceptual test has been used for purposes of pre-testing, as well as for purposes of post-testing, 
there is a potential testing effect threat to the internal validity of the research experiment. Concretely, the increase in 
conceptual test scores may be alternatively explained by the fact that students memorized pre-test items and their 
solutions which helped them to solve the post-test. However, this alternative explanation is not very probable because 
students were not told in advance that they will write the same test again. They were also not given the solutions of 
the pre-test. 

Another limitation of the research experiment is related to the fact that it failed to control the amount of time 
invested in the completion of homework. However, this is mostly due to the fact that, unlike students from the PP 
group, the students from the SB and VB groups were required to make observations and/or collect measurements. 
This required extra time but obviously was not crucial for constructing target knowledge, at least when it comes 
to conceptual understanding. The fact that SB and VB students exhibited more positive attitudes towards physics 
homework, besides the fact that they invested much more time in its completion is an important finding on its own. 


Conclusions and Implications 


Much of the physics learning may happen outside the physics classroom, particularly within the homework 
context. How effective this homework learning eventually will be, depends mostly on the quality of the mere home- 
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work assignment and teacher feedback. Often, physics homework boils down to solving end-of-chapter quantitative 
problems. Such an approach does not effectively target many of the important learning goals of modern-day physics 
education. Particularly neglected are science inquiry and affective learning goals, related to the development of posi- 
tive attitudes towards school physics. On the other hand, it is widely known that students’ attitudes towards school 
physics largely affect career choices and readiness for lifelong learning of physics. Therefore, it is important to identify 
alternative homework approaches that may be combined with paper-and-pencil homework to target a wider spectrum 
of learning goals. 

Findings from this research support the idea that homework learning may result in large improvements of 
conceptual understanding if it is carefully designed to target key cognitive processes from a given domain. Thereby, 
conceptual understanding is mainly developed through the cognitive processing of collected information about physi- 
cal phenomena, rather than through the mere activity of collecting measurements and doing observations. However, 
providing the students with the opportunity to learn by observing and collecting measurements positively affects their 
attitude towards school physics. Concretely, simulation-based homework and video-based homework proved to be 
superior to the paper-and-pencil homework when it comes to developing positive attitudes. In fact, simulation-based 
homework was found to be the most effective, which may be explained by the fact that students like to have an active 
approach to learning and to control the pace of their own learning. 

A practical implication of this research is that different formats of physics homework should be combined in the 
teaching practice. An advantage of the paper-and-pencil homework is that it is relatively easy to be prepared and 
can be similarly effective to simulation-based and video-based homework when it comes to developing conceptual 
understanding and problem-solving ability. However, to fulfill important science inquiry and affective learning goals, 
the homework format should be changed from time to time. One attractive option is to assign simulation-based in- 
quiry activities as homework. Compared to the video-based approach, the simulation-based homework assignment is 
much easier prepared and more effective in developing positive attitudes toward school physics, but equally effective 
in developing conceptual understanding. When assigning simulation-based homework, it is strongly suggested to 
accompany it with carefully designed worksheets which may offer important guidance in the absence of the teacher. 
The worksheets should be designed to facilitate students’ discovery of relationships, application of these relationships 
through predict-observe-explain activities and solution of physics problems situated within the context of simulations. 

In future research, it may be interesting to explore whether the effectiveness of simulation-based homework 
may be increased through learning by collaboration. Also, it would be interesting to conduct a carefully controlled 
experiment to compare the effectiveness of simulation-based learning in the classroom and homework setting. 
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Abstract. Online metacognitive skills 

are the real-time awareness of cognition, 
which can effectively promote science 
learning and improve performance in 
solving scientific problems. Therefore, it 

is important to enhance and diagnose 
students’ online metacognitive skills in 
science education. This study aimed to 
evaluate ninth-grade students’ online 
metacognitive skills while processing 
chemistry problems. To achieve this goal, 
this study constructed a framework for 
guiding the development of an instrument 
comprising 12 two-tier items. A total of 258 
ninth graders took part in the field testing 
in Jiangsu, China. A partial credit Rasch 
model analysis was employed to inform 
instrument development and evaluation. 
The results revealed that this instrument 
was valid and reliable for assessing 
students’ online metacognitive skills. 
Nearly 60% of the ninth-grade students 
in this sample were able to monitor their 
own thought processes or evaluate their 
own cognitive performance in processing 
chemistry problems. About one-third of 
the students could regulate their thought 
processes. However, less than 4% of the 
students could make attributions about 
their cognitive performance. 

Keywords: assessment instrument, 
problem-solving skills, Rasch measurement 
model, chemistry education 
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Introduction 


Metacognition refers to the knowledge and awareness of cognition (Fla- 
vell, 1979), indicating a person’s ability to reflect upon their own thoughts, 
experiences, and actions (Soto et al., 2018; Weil et al., 2013). Previous literature 
has proposed two types of metacognition: offline metacognition and online 
metacognition (De Clercq et al., 2000; Desoete, 2001). Offline metacognition 
is a general reflection on past or future activities (Violeau et al., 2020). In 
contrast, online metacognition is a mental process activated as the individual 
is processing the task at hand (Ng et al., 2021; Quiles et al., 2014). The term 
online, here, isa synonym for an ongoing situation where certain metacogni- 
tive functions take place (Mazancieux et al., 2021). 

In science education, online metacognition has been considered more 
important than offline metacognition, specifically in solving scientific pro- 
blems (Gilbert, 2005). The use of online metacognition can facilitate students’ 
abilities to actively control and regulate their thinking process for processing 
tasks (Efklides & Misailidi, 2010; Goldstein & Naglieri, 2011; Kuzle, 2018; Lavi et 
al., 2019; Seel, 2012; She et al., 2012). For instance, Cooper et al. (2008) found 
that the learners who performed well on chemistry problems scored high 
on measures of online metacognition. Rickey and Stacy (2000) also stated 
that online metacognition could compensate for lacking chemical problem- 
solving experience, and the students who were aware of and in control of 
their own thoughts were better able to solve chemistry problems than the 
students who were not online metacognitive. 

A large number of studies, however, have shown that students have lim- 
ited online metacognitive skills in solving chemistry problems, as measured 
primarily by observation, verbal protocols or self-reports (Mathabathe & 
Potgieter, 2017; Sandi-Urena et al., 2011; Wang, 2022). For instance, Pulmones 
(2007) discovered that many students were unaware of their thoughts, as 
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seen by their failure to monitor the progress of their work, determine whether their solutions were accurate, and 
explain why the problems were challenging. According to Bell and Volckmann (2011), at least half of the students 
overestimated their performance, and the worse they performed, the more biased they were. Teichert et al. (2017) 
noticed that students struggled with online metacognitive reflection on their problem-solving processes, and they 
had trouble explaining their problem-solving results. 

Therefore, there is an urgent need to develop students’ online metacognitive skills to help them to become 
successful problem solvers and obtain their expected learning outcomes. To achieve this goal, one essential step 
towards fostering students’ online metacognitive skills is to diagnose the extent of students’ online metacognitive 
skills in solving chemistry problems. However, to date, limited studies have focused on assessing students’ specific 
online metacognitive skills in solving chemistry problems (Schellings, 2011; Wang, 2015). In this regard, the pres- 
ent study aimed to evaluate ninth-grade students’ online metacognitive skills in solving chemistry problems. To 
fulfill this purpose, this study constructed a framework to guide the development of an instrument for identifying 
the specific online metacognitive skills that are challenging for students, which in turn would allow appropriate 
teaching aids to be provided. 


Literature Review 
Online Metacognitive Skills 


In science education, current literature has not achieved a consensus regarding the construct of online meta- 
cognitive skills (Desoete, 2001; Desoete & Roeyers, 2002; Efklides, 2002; Lee et al., 2019). According to Chen and 
Goverover (2021), online metacognition skills comprise the abilities of monitoring (e.g., identifying mistakes) and 
regulation (e.g., correcting responses). Similarly, Quiles et al. (2014; 2020) have identified two aspects of online 
metacognitive skills, including online metacognitive monitoring that involves the subjective self-assessment of 
confidence in answering the questions and online metacognitive control concerning the decision on whether to 
validate the answers. Sporer and Horry (2011) have noted that online metacognitive evaluation is an essential abil- 
ity to make estimates of the likely accuracy of decisions. In addition, researchers have also proposed that online 
metacognitive skills consist of students’ abilities to make attributions for the success or failure of their performance 
when confronted with problems (Wong, 2007) and their consciousness of task processing (Efklides et al., 1998; 
2008). In this sense, online metacognitive skills is an umbrella term encompassing the interrelated sub-skills nec- 
essary for solving problems. 


Online Metacognitive Skills in Solving Chemistry Problems 


Online metacognitive skills are critical in tackling chemistry problems (Wang, 2015). Students who employed 
more efficient online metacognitive strategies, such as using regulatory skills and reflective practices, could handle 
more complex chemistry problems (Cooper et al., 2008; Sandi-Urena et al., 2011). A study conducted by She et al. 
(2012) has shown that the top performers in solving chemistry problems were good at regulating their actions 
and evaluating whether the answers were correct via checking information. Teichert et al. (2017) found that ac- 
curate monitoring and reflection on ongoing mental processes could facilitate students’ application of cognitive 
resources within chemistry problem contexts. 

Specifically, Heidbrink and Weinrich (2021) proposed three iterative phases regarding the online metacogni- 
tive process in solving a chemistry problem. While solving a chemistry problem, students first identify the given 
information. They then deal with the information provided to ensure that it is working towards the goal, which 
involves being aware of some mistakes and debugging the errors. Finally, students reflect on the whole process. 
Similarly, Kipnis and Hofstein’s (2008) study has demonstrated that students judge whether the solutions to prob- 
lems are logical by carefully checking their actions and asking themselves if they are doing the right thing and if 
the results are reasonable. 


Assessing Online Metacognitive Skills 


In the field of chemistry education, students’ metacognitive skills have been assessed in a variety of ways. For 
instance, Cooper and Sandi-Urena (2009) developed a scale where students rated their degree of agreement with 
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27 metacognitive statements (e.g., | do not check if the answers are reasonable). Hawker et al. (2016) and Testa et 
al. (2023) asked students to estimate their performance after examinations and then computed the discrepancies 
between the actual and predicted scores. Gamby and Bauer (2022) evaluated metacognitive awareness and skills 
by examining students’ comments on their cognitive knowledge and beliefs. 

However, many researchers have warned that the previous measurements of metacognitive skills cannot 
assess online metacognitive skills because those measurements are unreliable, as they often fail to capture the 
dynamic metacognitive processes within a particular task (Jacobse & Harskamp, 2012; Wang, 2015). The validity 
of the online metacognitive data not gathered by online methods should also be questioned (Schellings, 2011). 

To evaluate students’ online metacognitive skills, some researchers have proposed so-called online mea- 
surement techniques (Bryce & Whitebread, 2012; Veenman et al., 2006), including think-aloud, observation, and 
eye-tracking methods administered to students during actual task performance (Bannert & Mengelkamp, 2008; 
Kinnunen & Vauras, 2010; Veenman, 2013; Winne, 2014). For instance, Dermitzaki (2005) conducted a direct obser- 
vation to investigate second graders’ self-regulative behaviour during the construction of a wooden vehicle or toy. 

Although online measurement techniques have shed light on detecting students’ online metacognitive 
skills, the majority of those techniques are labour- and time-intensive (Bannert & Mengelkamp, 2008; Veenman & 
van Cleef, 2019). For example, the think-aloud method requires well-trained raters to put a huge effort into scor- 
ing verbal protocols (Schellings et al., 2013). Observation of online metacognitive behaviour is time-consuming 
(Veenman & van Cleef, 2019) and often requires a substantial amount of effort to collect data, and the behavioural 
data must be examined by several observers according to a complex scoring system (McCord & Matusovich, 2019; 
Veenman, 2017). 

Compared with other online measurement techniques, questionnaires are the least labour-consuming 
method of acquiring and analyzing a massive amount of information (Schellings et al., 2013), and have therefore 
been widely used to measure online metacognitive skills. For instance, Lawanto (2010) developed the Engineering 
Design Project Inventory to measure university students’ online metacognitive skills in the process of engineering 
design. All data from the questionnaire were collected from participants during the engagement process. 

Specifically, two-tier items, consisting of one task and one question eliciting the online metacognitive activities 
behind the task response, have been one of the most popular types of questionnaire items for efficiently investi- 
gating online metacognitive skills (Dermitzaki, 2005). For example, Koren et al. (2005) and Quiles et al. (2014) used 
two-tier items (the Wisconsin Card Sorting Test, WCST) to evaluate online metacognitive monitoring and control. 
The participants first performed a matching task. Then, they were asked to rate their confidence in their responses, 
from which the extent of online metacognitive monitoring was measured. The participants were also requested 
to validate the task responses and decide whether the responses should count towards their overall grade, thus 
examining their online metacognitive control. 

In short, previous studies have made valuable contributions to investigating students’ online metacognitive 
skills. However, the currently articulated online metacognitive skills in solving chemistry problems have received 
too little attention in the research literature. While primary, upper-secondary, and university students were the main 
subjects in most of the existing related studies, the extent of lower-secondary school students’ online metacogni- 
tive skills in dealing with chemistry problems remains unknown. In order to address this gap, the present study 
aimed to develop and validate a measurement instrument by applying Wilson’s Construct Modeling Approach 
(Wilson, 2005). The findings of this study may provide a basis for further instruction regarding online metacogni- 
tion in solving chemistry problems. 


Research Purpose and Research Questions 


The current study intended to develop and validate a measurement instrument to assess ninth-grade students’ 
online metacognitive skills in solving chemistry problems. 
Specifically, two research questions were addressed: 
(1) What evidence supports the reliability and validity of measures of the instrument developed in this 
study for assessing students’ online metacognitive skills in solving chemistry problems? 
(2) Whatis the extent of ninth-grade students’ ability to monitor, evaluate, regulate, and make attributions 
of their own cognitive performance? 
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Theoretical Framework 


In reference to previous studies (Chen & Goverover, 2021; Quiles et al., 2020; Sporer & Horry, 2011; Wong, 
2007), this study defined online metacognitive skills as the ability to monitor, evaluate, regulate, and reflect on 
one's own performance in solving chemistry problems. In particular, four hierarchical levels comprise the construct 
of this study, which is hypothesized to delineate a set of complicated online metacognitive skills, from the least 
sophisticated (i.e., monitoring one’s own thought processes) to the most advanced (i.e., making attributions about 
one’s own cognitive performance), as shown in Figure 1. 


Figure 1 
Online Metacognitive Skills Framework 


Level 4 - Making attributions of one’s own cognitive performance 


) 
— SY : 
Level 3 - Regulating one’s own thought processes 


Level 2 - Evaluating one’s own cognitive performance 
yc : 
" 


| Level 1 - Monitoring one’s own thought processes 


The first level relates to students’ ability to monitor their thoughts. Monitoring has long been emphasized in 
scientific problem-solving literature (Hollingworth & McLoughlin, 2001; Rozencwajg, 2003). Monitoring processes 
frequently occur in the early stages of problem-solving and play essential roles in helping students modify their 
behaviour (Jacobse & Harskamp, 2012; Kuzle, 2013; Rickey & Stacy, 2000). Online metacognitive monitoring pro- 
cesses require students to concentrate on ongoing task performance, recognize if the thinking is clear, and identify 
whether the reasoning is correct or incorrect to ensure that the current thinking is going in the right direction. 

The second level concentrates on students’ ability to evaluate their cognitive performance, asking students 
to demonstrate the ability to assess their problem-solving processes, estimate the correctness and rationality of 
the solutions, and check their answers. Evaluation has been widely considered an essential element of online 
metacognitive skills that helps verify reasonable solutions to chemistry problems (Vo et al., 2022). Given that mak- 
ing performance judgments often occurs after and is dependent on students’ monitoring behaviour (Jacobse & 
Harskamp, 2012; Koriat, 2002), evaluating one’s cognitive performance is more demanding for students compared 
with monitoring (Azizah et al., 2019). 

The third level relates to students’ abilities to regulate their thought processes. Students who achieve this level 
are able to find and correct their mistakes, modify their operations, overcome their difficulties, and develop new 
solutions to deal with problems in an efficient way. As a critical component of online metacognitive skills (Rickey & 
Stacy, 2000), regulation is more advanced than monitoring or evaluating because it requires students to accurately 
monitor and assess their performance before they can direct their cognition (Cheng, 2011). 

The fourth level corresponds to students’ ability to make attributions of their own cognitive performance, 
requiring students to learn from their successes and failures by reflecting on their problem-solving processes and 
identifying the factors affecting their performance. Making attributions is not only important for solving chemistry 
problems (Zoller & Pushkin, 2007), but it can also influence judgment, self-regulation process, and achievement 
striving (Clifford, 1986; Lin, 2003; Ross, 1999). The ability to make attributions for their performance depends on 
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abilities such as monitoring, evaluation, and regulation (Garner, 2009; Zimmerman, 2011; 2013). Therefore, this 
ability is considered the most advanced in terms of online metacognitive skills. 


Research Methodology 
General Background 


Until now, few studies have focused on evaluating lower-secondary school students’ online metacognitive 
skills in processing chemistry problems. Therefore, the present study developed and validated an instrument to 
assess ninth-grade students’ online metacognitive skills in solving chemistry problems. The validity and reliability 
of the instrument were examined using Rasch analysis. Students’ online metacognitive skills were detected during 
the 2020-2021 academic year. 


Sample Selection 


Convenience sampling was used to recruit participants from two lower-secondary schools in Jiangsu, China. 
For the pilot test, 236 ninth graders (130 females and 106 males) were invited from one school to evaluate and 
refine the initial instrument. For the field test, 258 ninth graders (123 females and 135 males) were selected from 
the other school to confirm the revised instrument. The sample size was determined based on previous studies, 
which suggested that at least 200 participants would be adequate (Liu, 2020). These students were academically 
diverse and included low-, medium-, and high-performers. All the participants were volunteers and were assured 
of confidentiality. 


Instrument and Procedures 


The instrument development was guided by Wilson’s Construct Modeling Approach (2005) through six itera- 
tive steps: (1) developing a construct map; (2) generating items; (3) developing scoring rubrics; (4) administering 
a pilot test and revising items; (5) conducting the field test; and (6) Rasch analysis. 

Developing a construct map. A construct map specifies a continuum of skills that learners are expected to 
progress through for the target construct (Rittle-Johnson et al., 2011). In this study, a construct map (see Figure 1) 
for online metacognitive skills in chemistry problem-solving was developed. 

Generating items. In line with previous studies (e.g., Dermitzaki, 2005), a two-tier response format was ad- 
opted to generate the test items. The initial item pool was generated based on a comprehensive review of related 
measurements, including the Engineering Design Project Inventory (Lawanto, 2010) and the Wisconsin Card Sort- 
ing Test (Koren et al., 2005). A group of experts (three science education professors and two experienced science 
educators) and the authors took part in the development of the items. 

In particular, the following three tenets were used to review and revise the items in the initial pool: (1) each 
item should be linked to one level of skills, and all levels should have corresponding items. (2) The chemistry 
problems should be embedded in real-life contexts and could not be taken from the textbook, thereby avoiding 
the possibility of students solving problems by memorizing, and the problems must be cognitively challenging 
to promote students to think deeply and stimulate their online metacognitive thinking processes. (3) Given that 
the test aimed to assess online metacognitive skills, the knowledge of chemistry required should be reduced to 
a minimum. 

The initial version of the instrument included 20 two-tier items embedded in five tasks. An example task and 
details of the item illustrations are presented in Appendix A. 

Developing scoring rubrics. After the items were designed, scoring rubrics were constructed to evaluate 
students’ responses to the items. In this study, the scoring rubrics contained information about the performance 
indicators and codes for identifying the anticipated responses corresponding to the related level of performance. 
Among the initial 20 items, five items were scored from 0 to 1, and the other 15 items were scored from 0 to 2 
(examples are shown in Appendix B). 

Four graduate students in science education were recruited as raters to score each item independently. All 
the discrepancies that arose during the rating processes were resolved through several extensive discussions. The 
inter-rater reliability was determined as Cohen's kappa value of .95 (p < .01). 
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Administering a pilot test and revising pilot test items. To further refine the instrument, participants not 
only took part in the pilot test but also were interviewed afterwards to collect information regarding their experi- 
ences in the test. The results of the pilot test were analyzed using Rasch analysis to check the quality of the items. 
Based on the pilot test results, two tasks were removed because of low item discrimination, and three controversial 
items were modified following a focus group discussion with experts and student interviews. 

The final version of the instrument comprised 12 two-tier items. Table 1 shows the distribution of the items 
corresponding to the different levels of online metacognitive skills in solving chemistry problems. 


Table 1 
Field Item Distribution Regarding Online Metacognitive Skills in Solving Chemistry Problems 


Levels Performance expectations Items 

Level 1 Monitoring one’s own thought processes Q2/Q3; Q8/Q9; Q14/Q15 
Level 2 Evaluating one’s own cognitive performance Q2/Q5; Q8/Q11; Q14/Q17 
Level 3 Regulating one’s own thought processes Q1/Q4; Q7/Q10; Q13/Q16 
Level 4 Making attributions of one’s own cognitive performance Q5/Q6; Q11/Q12; Q17/Q18 


Conducting the field test. A 40-minute field test was conducted to ensure the validity and reliability of the 
revised instrument, as well as to determine the online metacognitive skills within the sample. The Rasch model 
was employed to analyze the field test data. 

Rasch analysis. In science education, Rasch measurement has been widely used for the rigorous development 
and examination of measurement instruments (Boone & Scantlebury, 2006; Planinic et al., 2019; Sideridis, 2007; 
Wang et al., 2017; Wren & Barbera, 2014). This study conducted Rasch analysis taking into account the following 
benefits: (a) the Rasch model is expressed both at the instrument level and the item level (Muller, 2005). (b) Rasch 
measurement focuses on the likelihood of the response rather than the answer per se (Wilson et al., 2006). (c) The 
features of the participants do not affect the item estimates (Chae et al., 2018). (d) Rasch measurement allows for 
converting the raw ordinal scores into interval equivalent scores expressed on a linear scale (Engelhard & Myford, 
2003). 


Data Analysis 


The development of the assessment instrument for this study and subsequent analysis of pilot and field test 
results were all guided by the application of Rasch measurement. Given that all the items in this study did not 
share the same scale steps, partial credit Rasch model analysis was applied for measurement development and 
instrument evaluation. All raw data were converted into Rasch measures (in logits) to generate interval data for 
further statistical calculations (Bond & Fox, 2015). Specifically, the person/item separation and person/item reli- 
ability indices were calculated to examine the instrument's reliability. Fit statistics and the person-item distribution 
map (Wright map) were used to evaluate the validity of the instrument. 


Research Results 


In presenting the findings, this study focused on the revised version of the measurement instrument. Data 
from the field testing were analyzed using Winsteps software (Linacre, 2009) version 3.72.3. 


Evidence for Reliability and Validity 


Unidimensionality. Unidimensionality is an explicit requirement for Rasch measurement (Smith, 1996), as- 
suming that the test measures only one underlying construct (Wind & Schumacker, 2021). This study performed a 
principal component analysis of residuals to check for unidimensionality (Bond & Fox, 2015). According to Linacre 
(2020), unidimensionality is achieved if the unexplained variance in the first contrast is less than two eigenvalues. 
The results indicated that the eigenvalue was 1.9 in the first contrast, thus supporting unidimensionality. 
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Reliability. In Rasch measurement, item reliability refers to the consistency of relative item measure location, 
and person reliability refers to the replicability of person placement across items (Bond & Fox, 2015; Linacre, 2020). 
The Rasch measurement model provides indices of item/person separation and item/person reliability like Cron- 
bach’s alpha for evaluating reliability across persons and items. The criteria for accepting reliability and separation 
in the Rasch model are above .70 (Fitzpatrick et al., 2004) and 2, respectively (Fisher, 2007). As Table 2 shows, the 
item reliability was high at .98, and the person reliability was .71. However, the item separation was acceptable at 
7.96, and the person separation was low at 1.55. 


Table 2 
Field Test Summary Statistics from Rasch Model Analysis 


Infit Outfit 
Measure Ero —__ Separation Reliability 
MNSQ ZSTD MNSQ ZSTD 
Person -0.24 0.61 1.00 -0.10 1.02 -0.10 1.55 a) 
Item 0.00 0.11 0.99 -0.50 1.02 0.20 7.96 .98 


Validity. Rasch fit statistics and the person-item distribution map (Wright map) are used to test the validity 
of the estimated measures (Bond & Fox, 2015). The Rasch measurement model generates a set of fit statistics to 
evaluate the extent to which each item matches the expectations of the Rasch model. This study reviewed Outfit/ 
Infit mean square residual (MNSQ) values and standardized mean square residual (ZSTD) values, with MNSQ values 
between 0.5 and 1.5 and ZSTD values between —2 and +2 as fit criteria for quality assurance (Bond & Fox, 2015). The 
point-measure correlation index (PTMEA) was also checked. The PTMEA refers to the Pearson correlation between 
the item score and the Rasch measure (Linacre, 2020), ranging from —1.0 to 1.0. The expected point-measure cor- 
relation value is positive, showing that the item-level scoring accords with the latent variable (Bond & Fox, 2015). 
As Table 3 shows, all the field items had acceptable MNSQ (ranging from 0.57 to 1.43) and PTMEA values (ranging 
from .39 to .63). However, nine of the twelve two-tier items had ZSTD values outside the fit range. 


Table 3 
Fit Statistics of 12 Two-tier Field Items from Rasch Model Analysis 


Infit Outfit 
a tesslins Model PT-MEASURE 
ae MNSQ ZSTD MNSQ ZSTD SURR: 
Q17/018 1.60 13 0.63 -5.6 0.65 -3.2 60 
Q11/Q12 1.46 13 0.57 -6.5 0.59 -4.4 63 
Q5/06 1.03 12 0.60 -57 0.76 -24 55 
Q7/Q10 0.51 2 1.26 28 1.43 3.8 40 
Q13/016 0.38 A2 1.20 22 1.37 3.4 39 
Q1/04 0.23 2 1.17 1.8 1.23 22 48 
Qs/att -0.28 12 0.80 -2.4 0.79 -2.4 54 
Q14/Q17 -0.51 12 0.94 -0.7 0.93 -0.8 53 
Q2/05 -0.61 12 0.96 -0.4 0.95 -0.6 44 
Qs/ag -0.99 2 1.31 35 1.28 3.2 55 
Q14/015 -1.20 2 1.11 1.3 1.09 11 46 
Q2/03 -1.61 2 1.31 3.8 1.29 35 60 
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The Wright map (person-item map) visually displays the relative difficulties of items and estimates of a per- 
son's ability on the same linear logit scale (Linacre, 2020). The left-hand side of the map locates the distribution of 
students’ ability measures, from the most capable (top) to the least capable (bottom). The right-hand side of the 
map presents the items from the hardest (top) to the easiest (bottom). In general, the mean of the item difficulty 
is centered at 0 logits. As Figure 2 shows, the 12 field items covered a range of item difficulty from —1.61 to 1.60 
logits. Ordered as hypothesized, the higher the items plotted on the Wright map, the higher the complexity level, 


thereby providing evidence of the construct validity of the scales. 


Figure 2 
Wright Map for Online Metacognitive Skills 
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Ninth-grade Student Online Metacognitive Skills 


To identify the extent of students’ online metacognitive skills, this study used the mean estimates of the 
item difficulty of each level as the cutoff value of each complexity level. As Table 4 shows, the four cutoff values 
are —1.27 logits for level 1, —0.47 logits for level 2, 0.37 logits for level 3, and 1.36 logits for level 4. These values 
increased along with the proficiency levels as hypothesized. On average, items at higher levels are more difficult 


than items at lower levels. 


In Rasch analysis, respondents have a 50/50 chance of correctly answering an item if they have an identical 
ability measure to the difficulty measure of the item (Boone et al., 2014). Therefore, students can be considered 
to have attained a particular level when they have the same ability measure as the cutoff value for this level. For 


example, a student with 0.0 logits reached Level 2 but had not yet achieved Level 3. 
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Items and Measures, and Cutoff Values of Levels 


Levels Items and measures Cutoff values 
Level 1 Q2/Q3 (-1.61), Q8/Q9 (-0.99), Q14/Q15 (-1.20) -1.27 
Level 2 Q2/Q5 (-0.61), Q8/Q11 (-0.28), Q14/Q17 (-0.51) -0.47 
Level 3 Q1/Q4 (0.23), Q7/Q10 (0.51), Q13/Q16 (0.38) 0.37 
Level 4 Q5/Q6 (1.03), Q14/Q12 (1.46), Q17/Q18 (1.60) 1.36 


As Figure 3 shows, overall, 14.34% of ninth-grade participants reached level 0, indicating that they were not 
able to utilize online metacognitive skills when solving chemistry problems. Of the participants, 81.78% were 
classified as Level 1 (27.13%), Level 2 (29.84%), or Level 3 (24.81%), suggesting that these students were not 
able to accurately evaluate, regulate, or make attributions of their problem-solving performance. Only 3.88% of 
participants achieved Level 4, meaning that they were able to successfully employ online metacognitive skills to 
complete chemistry problems. 


Figure 3 
Online Metacognitive Skills of Sample by Level 
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Discussion 


Following on from the paucity of research explicitly examining the online metacognitive skills of students dur- 
ing specific cognitive activities (Schellings, 2011; Wang, 2015), the goal of this study was to develop and validate an 
instrument for assessing ninth-grade students’ online metacognitive skills in chemistry problem-solving. Specifically, 
a construct map was developed to illustrate the different levels of online metacognitive skills, and an instrument 
comprising 12 two-tier items was created to measure the construct. The instrument's validity and reliability were 
verified using Rasch analysis. Eventually, the online metacognitive performance of ninth-grade students was tested 
and identified according to the assessment framework. In the following section, this paper discusses the quality 
of the instrument and the performance of the students in online metacognitive skills. 


Psychometric Properties of the Instrument 
The results indicated that the instrument had good psychometric qualities. The principal components analy- 
sis confirmed that the test measures only one underlying construct. The instrument's reliability was considered 


acceptable, with evidence that the indices of item/person reliability and item separation were fair. The validity of 
the instrument was checked using fit statistics and the person-item distribution map. 
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However, there are some aspects in which the instrument could be improved. First, the person separation 
was not perfect, indicating the limited ability of the instrument to distinguish high and low performers. This may 
be because the field test sample was from the same school and grade, which might mean that it was relatively 
homogeneous and not widely representative. Another reason could be the inadequate items to differentiate the 
sample, considering that each student was administered only three tasks. Some participants did not respond to 
all the questions, resulting in missing data in the study, which could reduce accurate measurement (Linacre, 2020). 
The test is a low-stakes test and the results are not used for decision making; the relatively low person separation 
in this study may not be a critical issue. 

The Wright map also presented several gaps in item coverage, suggesting a lack of items targeting some 
students. According to a widely accepted criterion, when the difficulty discrepancy between two adjacent items 
exceeds 0.5 logits, a gap is deemed substantively significant (Lai & Eton, 2002; Linacre, 2020). In this study, there 
were two obvious gaps between item Q8/Q6 (1.03 logits) and item Q7/Q10 (0.51 logits), as well as between item 
Q1/Q4 (0.23 logits) and item Q8/Q11 (—0.28 logits). Therefore, new items are needed to fill these gaps. Easier or 
harder tasks could also be designed to obtain more data about students who do exceptionally well or poorly (e.g., 
estimates of a person's ability under —1.61 or exceeding 1.60). 

Finally, nine items had ZSTD values outside the fit range. A ZSTD of more than 2 indicates underfit, while a 
ZSTD of less than —2 denotes overfit (Bond & Fox, 2015). Therefore, five items showed underfit, and four items 
showed overfit to the Rasch model. However, for application, MNSQ is more useful than ZSTD because the former 
demonstrates a practical data-model fit, whereas the latter presents a perfect data-model fit (He et al., 2022). The 
ZSTD values can be ignored if the MNSQ values are desirable (Boone et al., 2014; Linacre, 2020). Therefore, the nine 
items with acceptable MNSQ are not considered problematic. 

Overall, it was confirmed that the instrument assessing students’ online metacognitive skills was valid and 
reliable. 


Students’ Performance in Online Metacognitive Skills 


The findings of this study corroborate the work of Dermitzaki (2005) and Lawanto (2010). As the data showed, 
most students performed well in monitoring their performance during the problem-solving process, in line with 
previous literature (Azizah et al., 2019). The findings also echo Wang's (2015) study, which showed that monitoring 
commonly occurred among students who continually made judgments on the accuracy of their understanding 
and the effectiveness of their problem-solving strategies. Nearly 60% of the participants were able to evaluate 
their cognitive performance, similar to the findings of Wang's (2022) empirical research, which indicated that 
two-thirds of students reconfirmed the correctness of the solutions and checked their conclusions. However, this 
result differs from the findings of Overton et al. (2013), which revealed that a large portion of students had great 
difficulty assessing their problem-solving processes. Only one-third of the sample was able to regulate their thought 
processes successfully. This outcome is in agreement with other authors (Stevens et al., 2013) who noticed that 
many students changed their approaches to chemistry problems but did not achieve good outcomes. The ability 
to make attributions of their own problem-solving performance appeared to be the hardest to achieve. This may 
be because students rarely used effective reflective methods (Tu et al., 2020) and were confident in their results 
for problem-solving. Therefore, they made few causal attributions (Moller & Koller, 1999). 

This study showed the four hierarchical levels of ninth graders’ online metacognitive skills: monitoring, evaluat- 
ing, regulating, and making attributions. This agrees with existing studies (Koriat, 2002). As Kuzle (2013) reported, 
students were able to monitor their understanding of the problem, but they did not assess or direct their thinking. 
Azizah et al. (2019) also found that students had trouble using monitoring skills and that more of them struggled 
with evaluation skills. Wang (2015) observed that monitoring was more common than judgment, while reflections 
were occasionally absent in terms of frequency of occurrence. 

Overall, nearly 60% of the ninth graders in this sample were able to monitor their own thought processes 
or evaluate their own cognitive performance in processing chemistry problems, showing relatively good perfor- 
mance. This might be related to several factors. First, the field test participants were from one of the top schools 
in the city. In general, they have higher academic achievement than other students, which might result in better 
online metacognitive performance. This explanation makes sense because researchers have observed a positive 
correlation between academic outcomes and online metacognitive skills (Fleur et al., 2021; Treglia, 2018). Second, 
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due to cultural differences, Chinese students have high metacognitive sensitivity (van der Plas et al., 2022), which 
probably promotes the perception of cognitive processes. 

Some students, however, did not master online metacognitive skills, and their level of skills was low. The reason 
may be the absence of relevant experiences (Celik, 2022), inadequate metacognitive training (Veenman, 2012), 
and teachers’ lack of metacognitive knowledge (Heidbrink & Weinrich, 2021). Online metacognitive skills cannot 
be developed overnight (Pulmones, 2007). Therefore, there is an urgent need to give students more opportunities 
to practice online metacognitive activities, and teachers should improve their online metacognitive awareness. 


Conclusions, Implications, and Limitations 


In light of the absence of research to measure lower-secondary school students’ online metacognitive skills, 
this study developed an instrument for the evaluation of ninth-grade students’ online metacognitive skills in solving 
chemistry problems. Rasch analysis suggested that the instrument functioned as expected. The unidimensionality, 
fit statistics, Wright map, reliability and separation estimates verified the validity and reliability of the instrument. A 
sample of 258 Chinese ninth-grade students was measured and assessed using the instrument. Results showed that 
students’ online metacognitive skills progressed along with their proficiency levels as hypothesized (i.e., monitoring, 
evaluating, regulating, and making attributions). However, only a few students were able to successfully employ 
online metacognitive skills to complete chemistry problems, while most students could not accurately monitor 
their cognitive processes, judge their task performance, regulate their thought processes, or make attributions of 
their chemistry problem-solving outcomes. 

Researchers could find some inspiration from this work to generate a trustworthy measurement instrument. 
Guided by the Construct Modeling Approach, this study created an assessment tool using a more specific and 
detailed procedure that included building a construct map, generating items, developing scoring rubrics, admin- 
istering a pilot test, and revising test items. Furthermore, this research may contribute to a better understanding 
of students’ online metacognitive skills when they deal with problems and difficulties. Test designers can develop 
items to measure students’ online metacognitive skills to offer valuable guidance and information to teachers. In 
addition, the instrument could serve as an available tool to enable teachers to identify students’ current level of 
online metacognitive skills, change teaching strategies promptly, and determine the most effective educational 
interventions, eventually leading to better chemistry learning outcomes. 

Some limitations must be considered. First, because of the limited conditions, the sample size was small, and 
all participants were from ninth grade in Jiangsu, China, which affected the statistical power of the analysis con- 
ducted. Second, convenience sampling may have reduced the generalizability of the results. Third, since all data 
were generated in chemistry problem-solving settings, the results may not generalize to other disciplines (e.g., 
biology, physics, earth science, engineering). Finally, this study did not conduct a think-aloud study to confirm 
whether participants did or did not engage in online metacognitive activities as intended. 

Therefore, future research may need to include more diverse samples and more discriminative items to increase 
the statistical power of analyses. Further study may use a mixed-method research design to gain deeper insights 
and investigate the factors influencing students’ online metacognitive skills. Finally, researchers could explore the 
connection between online metacognitive skills and chemistry problem-solving abilities, and infer whether online 
metacognitive activities may have a positive effect on chemistry problem-solving. 
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Appendix A. An example task 


CRYSTALLIZATION 


Potassium nitrate partially precipitates in the form of crystals when cooling a hot concentrated aqueous solution of potassium nitrate. There 
are many cooling methods, such as rapid cooling, slow cooling, and variable speed cooling. Different cooling rates affect the degree of 
crystallization. A student used three cooling methods to crystallize the potassium nitrate, and the crystal particle size distribution is shown in 
the figure below. Which cooling method should be chosen to obtain large crystals with relatively uniform particle sizes? (Do not answer this 
question here. Please answer the following questions in sequence.) 


60 
; : A 
os —#—rapid cooling method aN 
*-@- slow cooling method i ‘ 
-4- variable speed cooling method ‘ ‘ 
40 } , . 


30 F 


mass fraction / % 


20 F 


200 400 600 800 1000 1200 
crystal particle size / um 


Q13. Before answering the above question, please estimate the probability that you can answer the question correctly. Tick “V” in the table 
below according to your judgment. 


incorrect correct 


Q14. According to the figure above, which cooling method should be chosen to obtain large crystals with relatively uniform particle sizes? 
Please explain the thinking processes and ideas you used to come to your answer. 


Q15. Have you recognized if the thinking processes are clear and whether the reasoning is correct or incorrect when solving this chemistry 
problem? 


A. Yes. My thinking is clear and my reasoning is correct. 
B. Yes. My thinking is not clear, and | encounter difficulties in reasoning about this problem. 
C. No. | do not have such recognition. 


Q16. Based on your answers in Q14, please re-estimate the probability that you can answer this question correctly. Tick “v’ in the table 
below according to your judgment. 


incorrect correct 


Q17. Please evaluate the answer you gave to Q14? 
A. My answer is correct. 

B. My answer is partially correct. 

C. My answer is incorrect. 

D. It is hard for me to evaluate my answer. 

Q18. Why did you make the evaluation in Q17? 


Item Q14 and item Q15 (abbreviated to Q14/Q15) were designed in accordance with Level 1, which assessed 
whether students monitored their own thinking during the process of solving a chemistry problem. The first tier 
required students to answer a question about which cooling method should be chosen to obtain large crystals 
with relatively uniform particle sizes and to write down their thinking processes. The second tier contained three 
options that depicted different monitoring awareness about the thinking processes when solving the first-tier 
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question and required students to choose one that best fitted their situation. The response demands of Q14 and 
Q17 (abbreviated to Q14/Q17) were in line with Level 2, which assessed online metacognitive evaluation. The sec- 
ond tier consisted of four options from which students had to choose one to indicate their evaluation results on 
the answers they gave in the first tier. Q13 and Q16 (abbreviated to Q13/Q16) were generated to measure online 
metacognitive regulation (Level 3). Immediately after reading the scenario and question, students were required 
to estimate the likelihood of giving an accurate response to the question in the first tier. After writing down their 
answers, respondents were asked to rate the probability again in the second tier. The difference between the two 
probability values was used to infer whether students might have regulated their thinking and thus changed their 
awareness about whether the problem could be solved. The Q17 and Q18 (abbreviated to Q17/Q18) belonged to 
Level 4, examining the ability of students to make attributions of their own chemistry problem-solving performance. 
Students evaluated their problem-solving performance in the first tier, and then they were provided with a space 
to write down the reasons for their evaluation in the second tier. 


Appendix B. The scoring rubrics 


Level Items Score Performance Evaluation criteria 


The first-tier responses are correct, and the second-tier re- 
sponses are option A. 


romelpants Gal aceulaieyy The first-tier responses are partially correct, and the second-tier 


2 monitor their own thought pro- ; 
iscaee responses are option B. 
: The first-tier responses are incorrect, and the second-tier 
responses are option B. 
Q2/Q3 Q8/Q9 Q14/ Be eee : 
eyel'4 Q15.Q20/021 Q26/ he first-tier responses are correct, and the second-tier re- 
- ; .. sponses are option B. 
Q27 Participants can monitor their =... ; : 
he first-tier responses are partially correct, and the second-tier 
1 own thought processes, but not : 
aeearital weneudh responses are option A. 
y on The first-tier responses are incorrect, and the second-tier 
responses are option A. 
0 Eaenalts Cannon Moana The second-tier responses are option C. 


their own thought processes. 


The first-tier responses are correct, and the second-tier re- 
sponses are option A. 

The first-tier responses are partially correct, and the second-tier 
responses are option B. 

The first-tier responses are incorrect, and the second-tier 
responses are option C. 


Participants can accurately 
2 evaluate their own cognitive 
performance. 


The first-tier responses are correct, and the second-tier re- 

sponses are option B. 

The first-tier responses are correct, and the second-tier re- 
Q2/Q5 Q8/Q11 sponses are option C. 

Level2 Q14/Q17 Q20/Q23 The first-tier responses are partially correct, and the second-tier 
Q26/Q29 responses are option A. 

The first-tier responses are partially correct, and the second-tier 

responses are option C. 

The first-tier responses are incorrect, and the second-tier 

responses are option A. 

The first-tier responses are incorrect, and the second-tier 

responses are option B. 


Participants can evaluate their 
1 own cognitive performance, but 
not accurately enough. 


Participants cannot evaluate 
0 their own cognitive perfor- The second-tier responses are option D. 
mance. 
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Level Items Score Performance Evaluation criteria 


Participants can successfully 
regulate their own thought 

2 processes, which makes their 
problem-solving performance 
better. 


The probability rated by participants in the second tier is higher 
than the probability rated in the first tier. 


Participants cannot success- 


ae) ee fully regulate their own thought 


Level3 =. Q13/Q16 Q19/Q22 The probability rated by participants in the second tier is equal 


Q25/Q28 : sia haaas maintaining ue to the probability rated in the first tier. 
same problem-solving perfor- 
mance. 
Participants cannot regulate 
0 their own thought processes, The probability rated by participants in the second tier is lower 
which makes their problem- than the probability rated in the first tier. 
solving performance worse. 
rae a vane alia Reasonable explanations for the problem-solving performance 
1 tions of their own cognitive Pieiien : 
are given in the second tier. 
Q5/Q6 Q11/Q12 performance. 
Level4 = Q17/Q18 Q23/Q24 
Q29/Q30 Participants cannot make at- 
here i sp Unreasonable explanations for the problem-solving performance 
0 tributions of their own cognitive ee ; 
are given in the second tier. 
performance. 
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Abstract. Scientific and technological 
research is important to everyone because 
their results affect humanity worldwide. 
Although science and technology bring 
many benefits to humanity, individuals 
and societies can face the negative 
consequences of science and technology. 
To prevent and solve the negative 
consequences of research in these fields, 

it is important to educate scientists and 
engineers to have social responsibility. 
However, no research has addressed pre- 
service science teachers’ views of social 
responsibility in literature. Therefore, 

this study aimed to explore Chinese 
pre-service science teachers’ views on 

the social responsibility of scientists and 
engineers. The research participants were 
194 pre-service science teachers. A social 
responsibility instrument was administered 
for data collection. The results showed 

that participants had high mean scores 

in five areas of social responsibility anda 
good awareness of the role of scientists 
and engineers in addressing problems and 
risks related to human, environmental, and 
societal impacts. However, they had lower 
mean scores in three areas requiring action 
or involvement. In addition, the results did 
not show statistical differences by gender 
or grade level. In light of our findings, 

we conclude pedagogical and practical 
implications for teachers and researchers in 
teaching social responsibility. 

Keywords: pre-service science teachers, 
science education, social responsibility, 
social responsibility of scientists and 
engineers, quantitative research 
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CHINESE PRE-SERVICE 
SCIENCE TEACHERS’ VIEWS OF 
THE SOCIAL RESPONSIBILITY 
OF SCIENTISTS AND 
ENGINEERS 


Shen Zhang 


Introduction 


The progress of research in science and technology is important to ev- 
eryone because every person living in the world is affected by the results of 
science and technology. Although research in science and technology brings 
many benefits and advantages to humanity, individuals and societies may face 
negative consequences (Tonn & Stiefel, 2019). To minimize and prevent the 
negative consequences of research in these fields, it is important to educate 
engineers and scientists to be socially responsible (Bielefeldt & Canney, 2016; 
Ko et al., 2023; Varma, 2000). Social responsibility derives from the “ethics of 
theory” concept first proposed by Noddings (1984). Specifically, the ethics 
of theory refers to “..the willingness and capacity to take responsibility that 
“something” is done to provide for the need in question.’(Sevenhuijsen, 2003, 
p. 21). Thus, this theory consists of attending to the needs of someone and 
something. In today’s world, people and countries must have engineers and 
scientists to solve the more complex problems faced by people and societies, 
such as the coronavirus-19 epidemic, environmental pollution, dangerous 
chemicals, and global warming (Ambusaidi & Al Fulaiti, 2022; Kirby, 2021; 
Nkaizirwa et al., 2021). Considering that scientists and engineers use their 
unique knowledge and skills to solve the most complex problems, they have 
an increased responsibility to benefit humanity and protect it from the bad 
consequences of unprecedented developments such as COVID-19 (Bielefeldt 
& Canney, 2016; Buie, 2017; Canney & Bielefeldt, 2015; Kobylarek, 2019; Sakha- 
rov, 1981; Zandvoort, 2007). Having more socially responsible graduates in 
science, technology, mathematics, and engineering (STEM) fields requires 
understanding the societal impacts of scientific and engineering research 
(Bielefeldt & Canney, 2016; Godhade & Hundekari, 2018; Ko et al., 2022). 

In the literature on science education, much more research has been 
conducted over the past three decades, using the “views on science, tech- 
nology, and society (VOSTS)” instrument developed by Aikenhead and Ryan 
(1992) to examine the social responsibility of pre-service science and science 
teachers. However, the VOSTS instrument included a few items regarding 
being socially responsible (Ko et al., 2023). It is generally accepted that since 
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the emergence of COVID-19, the importance of scientists and engineers using their knowledge and skills to ben- 
efit society seems to have increased at a higher level. Therefore, it is necessary to emphasize the importance of 
promoting the social responsibility of scientists and engineers. In this parallel, it is important to research the views 
of individuals who will implement the teaching profession in STEM fields. In particular, it is a revealing question 
to what extent the students in teacher education programs are aware of the social responsibility of scientists and 
engineers. However, in the research literature, only a few studies have examined pre-service teachers’ views of sci- 
entists and engineers. For example, a study by Ko et al. (2022) explored STEM undergraduate majors’ views on the 
social responsibility of scientists and engineers in South Korean universities. Their results showed that the average 
scores for three factors (HUMAN, ENVIR, and CONSEQ) were relatively higher than the other five factors (NEEDS, 
COMGOOD, CIVIC, COMMU, and POLICY). Their cluster analysis revealed five distinct groups with similar patterns 
of social responsibility. Another study by Lee et al. (2022) studied the views of PSTs on the social responsibility of 
scientists and engineers on socioscientific issues and a problem-based inquiry approach. Their results showed that 
pre-service science teachers’ views on social responsibility had changed statistically significantly after the project. 
The pre-service science teachers seriously engaged with the social responsibility of scientists and engineers by 
studying science and technology and engaging in problem-solving and action. Specifically, the researchers found 
that PSTs agreed to address societal needs and demands, pursue the common good, use their expertise to engage 
and help the community, communicate about potential risks, and participate in policy-making. In addition to these, 
researchers have conducted some studies to explore and understand the views of engineering students and gradu- 
ates in STEM fields (Bielefeldt & Canney, 2016; Canney & Bielefeldt, 2015). In general, the results of these studies 
showed that engineering students in STEM fields accepted the importance of engineering skills to impact society 
positively and have an awareness of helping people in need. However, the results showed that the awareness of 
participating in science and technology research is lower (Canney & Bielefeldt, 2015). 

A review of existing studies on social responsibility in science education reveals that very little information 
about social responsibility is known to scientists. However, much is not known about prospective science teach- 
ers’ understanding of the social impact of scientific and engineering research and what factors are effective in 
understanding social responsibility from the perspective of pre-service teachers who will teach future STEM 
majors in schools. Only one study on social responsibility examined prospective science teachers’ views (Lee, Ko, 
& Hong, 2022). Therefore, there is a need to examine how PSTs think about social responsibility. For this reason, 
the present study aimed to examine pre-service science teachers’ views on the social responsibility of scientists 
and engineers. The research question guiding this study was: What are pre-service science teachers’ views on the 
social responsibility of scientists and engineers? The present study helps to fill a research gap in the literature by 
providing new insights for educators and scientists and a better understanding of views on social responsibility. 


Research Methodology 
Design 


The present study was conducted using a quantitative research design to answer the research question. The 
quantitative research design, as Creswell states, “employs strategies of inquiry such as experimental and surveys and 
collects data predetermined instruments that yield statistical data” (Creswell, 2003, p. 18). The nature of this research 
design includes collecting quantitative data on several variables studied to answer the research question. Research- 
ers (Cohen & Manion, 1994; Creswell, 2003) have pointed out that research that uses questionnaires or surveys 
to collect data can be classified as quantitative research design. Quantitative data in this research were collected 
using an instrument developed by Ko et al. (2023). The independent variables were gender and grade level, and 
the dependent variable was pre-service science teachers’ views about the social responsibility of scientists and 
engineers. The study was conducted over two weeks in October 2022. The participants who enrolled in the Faculty 
of Education at a public research university in Beijing, People’s Republic of China, were invited to the research. 
Under the scope of this study, the participants were invited through university professors who gave courses at the 
faculty. Two hundred fifty copies of the instrument consisting of thirty items were administered to the participants. 
In this study, random sampling was used. All of the participants agreed to participate voluntarily in the research. 
Of the 228 copies answered by the participants, 194 valid questionnaires were responded to by the participants, 
with a retrieval rate of 86%. The participants were enrolled in the Department of pre-service science teacher educa- 
tion. While collecting the data, the participants’ demographic information and their responses to the instrument's 
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items were asked. The participants did not write their names and last names while responding to the items. All the 
participants had the opportunity to give their responses freely. 


Participants 


A total of 194 PSTs were invited to participate in the present study because they were easily accessible and 
convenient sampling for research and also had some characteristics necessary for the research, such as being PST 
and belonging to different gender and grade levels to allow for comparison across genders and grade levels (Cre- 
swell, 2003). The participants’ demographic information is presented in Table 1. Of the participants, 52.6% were 
male, while 47.4% were female. Most participants were in the fourth grade at Tsinghua University in 2022 (53.6%). 
The percentages of other grades in 1, 2, and 3 were 15.5%, 19.1%, and 11.9%, respectively. The participants received 
courses about the history of science and technology, science-technology-society, and science communication 
and museum. 


Table 1 
The Participants’ Demographic Information 


n % 

Male 102 52.6 
Gender 
Female 92 47.4 
1 30 15.5 
2 37 19.1 
Grade Level 

3 23 11.9 
4 104 53.6 
Total 194 100.0 


Data Collection Instrument 


To collect data in this study, an instrument developed by Ko et al. (2022) was used to measure participants’ 
views on the social responsibility of scientists and engineers. Ko et al. (2022) named the scale: “Views of Social 
Responsibility of Scientists and Engineers (VSROSE)” The instrument consists of thirty (30) items and eight factors. 
These factors are referred to by the researchers as (1) concern for human welfare and safety, (2) concern for envi- 
ronmental sustainability, (3) consideration of societal risks and consequences, (4) consideration of societal needs 
and demands, (5) pursuit of the common good, (6) civic engagement and services, (7) communication with the 
public, and (8) participation in policy decision making. 

When Ko et al. (2022) developed the VSROSE, they aimed to include all aspects of social responsibility based 
on the ethics of theory. The studies of Ko et al. (2022) resulted in a valid and reliable instrument to measure people's 
views on the social responsibility of scientists and engineers. The VSROSE is the most recent instrument developed 
in the literature to cover a broader spectrum of social responsibility. 

After using the VSRoSe instrument in this study, a reliability analysis and calculated Cronbach's alpha value for 
thirty items were performed. For all items used in this study, obtained a Cronbach's alpha value of .88 was found. 
This value suggests that the data collection instrument is reliable for data collection and analyses. 


Procedure 
To use the VSROSE instrument in this study, a bilingual science educator fluent in Chinese and English trans- 


lated it into Chinese. Following this translation, another science educator fluent in both languages translated the 
instrument from Chinese to English. Then the researchers asked three scientists who teach courses in the same 
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department to review and rate the original and back-translated versions of the instrument to compare and assess 
the comprehensibility of the language and items. They gave the researchers some minor feedback on the words 
and items. Once the experts’ feedback on the translated and back-translated versions was available, the research- 
ers finalized the final version of the instrument and began to use it in the research. The VSROSE instrument was 
administered to prospective teachers in October 2022. A total of 194 students responded to the instrument with 
surveys, with a response rate of 86%. Two researchers asked pre-service teachers and their professors to assist 
participants in responding to the data collection instrument. The researchers informed the participants of the 
purpose and scope of the survey and guaranteed that their responses would not be shared with third parties. Re- 
searchers answered participants’ questions about the research and the elements of the data collection instrument. 
The researchers found that participants could answer the instrument in 20 to 40 minutes. Participants answered 
the instrument in their classroom at the faculty of education. To answer the research question, participants were 
limited to prospective teachers. 


Data Analysis 


We performed descriptive statistical analyses for the instrument by performing measures and normality checks 
by skewness, kurtosis, mean, and standard deviations to generate the results. Table 2 presents the results regarding 
the descriptive statistics. We checked all assumptions for normality and parametric analyses during the descriptive 
analysis. After performing the analysis, the results revealed a normal distribution and the option of performing 
parametric analyses. In addition, we performed a t-test to understand the differences between the mean scores of 
male and female participants. Additionally, a one-way ANOVA analysis was conducted to identify significant and 
nonsignificant differences between grade levels, and Cronbach's alpha value to test the instrument's consistency 
and reliability in this study. Because Cronbach’s alpha value (Cronbach's alpha value was 0.85) is higher than > 0.7, 
the instrument used in the study was appropriate for this study. All statistical analyzes were performed using the 
SPSS 29.0 data editor. 


Research Results 


Table 2 shows the descriptive results of the participants’ views of the social responsibility of scientists and 
engineers. The results revealed that the participants’ views were within a specific range. Their views vary in range 
from 3.87 to 4.12. The results also revealed that the participants had a good awareness of concern for environmen- 
tal sustainability (ENVIR), civic engagement and service (CIVIC), consideration of societal risks and consequences 
(CONSEQ), communication with the public (COMMU), and the pursuit of the common good (COMGOOD), while 
concern for human welfare and safety (HUMAN), Consideration of societal needs and demands (NEEDS), and 
participation in policy-making (POLICY) were the lowest. In sum, mean scores for interest in and consideration 
of human, environmental, and social impacts (ENVIR and CIVIC, CONSEQ, COMMU, and COMGOOD) tended to be 
higher than mean scores for areas directly related to social practice (NEEDS and POLICY). 

Regarding the individual subfactors, the HUMAN area consisted of items about social responsibility in scientific 
and technological research and the impact of the results on human health and safety. All items in this area gener- 
ated moderate awareness of the impact of scientific and technological research on humans. The average scores 
ranged from 3.79 to 4.15 points (See Table 3). In particular, item five, “Prevent humans from the risks at the least,’ 
received the highest mean scores compared to the first four items. This result shows that the participants agreed 
with using science and technology to prevent humans from facing risks. 


Table 2 
The Descriptive Statistics Results 


Number 


: Min. Max. M SD Skewness Kurtosis 
of items 
HUMAN 5 1.60 5.00 3.87 0.76 -.766 410 
ENVIR 3 2.33 5.00 4.12 0.59 -.507 -.155 
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Pe Min. Max. M SD Skewness Kurtosis 
CONSEQ 5 3.00 5.00 4.05 0.49 -.265 -.804 
NEEDS 3 2.00 5.00 3.91 0.72 -.648 -.180 
COMGOOD 3 3.00 5.00 4.09 0.54 -.346 -117 
CIVIC 5 2.40 5.00 4.12 0.57 -.139 237 
COMMU 3 2.33 5.00 4.11 0.72 -.436 -.890 
POLICY 3 1.33 5.00 3.93 0.79 -.593 .057 


The second subfactor, consideration of the sustainable environment (ENVIR) area, included the responsibility of 
scientific and technological research to minimize various risks to the ecosystem and environment. The participants 
rated the items in this area with higher mean scores. All items in this factor had a high average score of 4.11 to 
4.15. This result means that the participants approve of using science and technology to promote the ecosystem 
and environment and increase sustainable development. 

The third subfactor, consideration of societal risks and consequences (CONSEQ), included five items about 
identifying and recognizing social problems during science and technology research. These five items received 
high recognition ranging from 3.96 to 4.19 points. The results demonstrated that the pre-service teachers support 
science and technology to avoid societal risks and problems. 

Regarding the fourth subfactor, the Social Needs and demands (NEEDS) area consisted of three items. The 
items in this area are related to the responsibility of scientists and engineers for the needs of society in scientific 
and technological research. These items’ mean scores are between 3.78 and 4.10 points. One statement, item 15, 
asked about conducting research following society's values and expectations, received a lower score than the 
other two items in this factor. 

Results on the fifth subfactor, the pursuit of the common good (COMGOOD) area, included three items that 
asked about the purpose of research to promote human life and welfare. Mean scores ranged from 3.98 to 4.18 on 
this factor. For example, the 17th item, emphasizing that research can improve the quality of human life, received 
a good score (4.13 points). Another item (18), which asks about promoting human welfare and safety as the main 
goal of one’s research, received a better average score (4.18 points). This result means that participants approve of 
the scientist and engineers’ responsibility to foster the quality of people's lives. 

The sixth subfactor, civic engagement and services (CIVIC) included five items related to the responsibility 
of scientists and engineers to solve societal problems related to science and technology. The mean scores of the 
items in this factor ranged from 4.01 to 4.26 points. Item 23, which asked about collaborating with knowledgeable 
and interested citizens to solve problems related to science and technology, had the lowest score in this factor. 
Another item, 20, which asked about willingness to participate in civic affairs in solving problems related to sci- 
ence and technology, had the highest mean scores in this domain. Communication with the public (COMMU), 
the seventh subfactor of the VSRoSE instrument, contained three questions that asked about the responsibility of 
scientists and engineers to communicate with the public, to use simple language that all people can understand, 
and to explain to the public knowledge and research needed to solve societal problems. The result shows that all 
three items received a good average score between 4.11 and 4.12. 

The last subfactor, participation in political decision-making (POLICY), had the lowest scores, ranging from 
3.84 to 4.13. The items in this area asked about participation in the policy-making process related to science and 
technology. In particular, item 29, which refers to active participation in the policy-making process related to sci- 
ence and technology, received the lowest recognition in the survey (3.85 points). Table 3. Descriptive statistics of 
VSROSE responses according to items. 
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Factors Items M SD 
Concem for Human Wel- 1. Not harm human health at the least 3.81 0.99 
fare and Safety HUMAN 

2. Place utmost importance on human health 3.79 1.00 
3. Be vigilant whether his/her research risks human safety 3.82 0.98 
4. Consider the possible adverse effects on human health 3.81 0.99 
5. Prevent humans from the risks at the least 4.15 0.79 
Concern for Environmen- — . Protect the environment during the research process 4.15 0.80 
tal Sustainability ENVIR 
7. Minimize the effects on the ecosystem 4.12 0.84 
8. Promote sustainable development in the environment 4.11 0.83 
Consideration of Societal 9. Recognize the potential social problems in one’s area of expertise values of multiple 3.96 O71 
Risks and Consequences _ stakeholders : . 
CONSEQ 
10. Be able to identify social problems inherent in modern science and Technology 4.19 0.73 
11. Be cognizant of the contribution that one’s work can make to the advancement of the field 4.09 0.76 
12. Be able to identify pressing social problems in one’s area 4.00 0.76 
13. Carefully examine the conflicting 4.03 0.91 
Consideration of Societal 14. Consider whether one’s research generates knowledge needed by the society 4.10 0.99 
Needs and Demands 
NEEDS 15. Conduct research consistent with the values and expectations of the society 3.78 1.08 
16. Identify the societal needs and expectations for scientific and engineering research 3.87 0.95 
Pursuit of the Common 17. Conduct research that can enhance the quality of human life 4.13 0.74 
Good COMGOOD 
18. View the promotion of human welfare and safety as a primary goal of one’s research 4.18 0.72 
19. View reducing the challenge that people experience in their daily life as an important goal 3.98 0.76 
of one’s research , ‘ 
Civic Engagement and 20. Be willing to participate in civic affairs if the goal of the affair is to solve the problems 
Services CIVIC related to science and technology 4.26 0.71 
21. Collaborate with the general public and citizens to solve the problems related to science 4.10 0.75 
and technology ‘ : 
22. Actively encourage others to participate in solving problems related to science and technol- 414 0.75 
ogy 
23. Collaborate with knowledgeable and interested citizens to solve problems related to sci- 401 0.79 
ence and technology , : 
24. Serve an advisory role for the public in their area of expertise 4.10 0.73 
Communication with the 25, Make the public familiar with science using media such as books, articles, blogs, and At 081 
Public COMMU lectures : ; 
26. Explain the knowledge and research necessary for solving social problems to the public 4.11 0.81 
27. Knowledge or research regarding science and technology should be explained in a way 412 0.85 
that is easy for the general public to understand ; ; 
Participation in Policy 28. As a member of a professional organization of scholars, one must influence the policy- : 
ai . : : 4.13 0.87 
Decision Making POLICY — making process related to science and technology 
29. Actively participate in a policy-making process related to science and technology 3.85 0.92 
30. Emphasize its importance and this must attract investment in science and technology 3.84 0.94 
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Table 4 shows the results of the mean scores of male and female prospective teachers on the social respon- 
sibility of scientists and engineers. Accordingly, male participants scored higher on the COMMU, CIVIC, ENVIR, 
CONSEQ, and COMGOOD subfactors. They scored lower on the subfactors HUMAN and POLICY. They had the 
lowest mean score on the subfactor POLICY. On the other hand, female participants had higher scores on ENVIR, 
CIVIC, COMMU, COMGOOD, and CONSEQ, POLICY. However, females had the lowest scores on the subfactors HU- 
MAN and NEEDS. In addition, we found the lowest mean scores for males and females in the HUMAN area. As can 
be seen from the mean scores in Table 4, the results of the t-test showed no significant differences between all 
subfactors of the instrument. 


Table 4 
Comparison of Factors According to Gender 


Male Female 
Factors ee t p 
M SD M SD 
HUMAN 3.88 0.77 3.87 0.76 .132 895 
ENVIR 4.09 0.58 4.16 0.60 -.799 425 
CONSEQ 4.02 0.48 4.08 0.49 -.749 455 
NEEDS 3.95 0.63 3.86 0.80 836 404 
COMGOOD 4.10 0.55 4.09 0.54 .090 928 
CIVIC 4.08 0.60 4.16 0.53 -1.056 292 
COMMU 4.16 0.74 4.05 0.70 1.138 257 
POLICY 3.87 0.85 4.01 0.72 -1.183 238 


Table 5 shows the results of a one-way ANOVA analysis to compare the mean scores in each subfactor ac- 
cording to grade levels among the PSTs. The results with students’ grade levels show no significant differences in 
all sub-factors. The average scores in all grades revealed no significant differences. 


Table 5 
One-Way ANOVA Results for Comparison According to Grade Level 


ANOVA 
Ss df Ms F p 
Between Groups 108 3 .236 
Human Within Groups 112.509 190 592 399 .754 
Tota 113.218 193 
Between Groups 1.159 3 386 
Envir Within Groups 66.816 190 352 1.098 351 
Tota 67.975 193 
Between Groups 1.169 3 390 
CONSEQ Within Groups 45.454 190 239 1.629 .184 
Tota 46.624 193 
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ANOVA 
Ss df MSs F p 
Between Groups 2.353 3 .184 
NEEDS Within Groups 98.383 190 518 1.515 212 
Tota 100.736 193 
Between Groups 1.040 3 347 
COMGOOD Within Groups 56.988 190 .300 1.156 328 
Tota 58.028 193 
Between Groups 1.743 3 581 
CIVIC Within Groups 61.626 190 324 1.791 .150 
Tota 63.369 193 
Between Groups 1.406 3 469 
COMMU Within Groups 101.210 190 533 .880 452 
Tota 102.616 193 
Between Groups 442 3 147 
POLICY Within Groups 121.412 190 639 .230 875 
Tota 121.854 193 
Discussion 


This study aimed to explore Chinese PSTs’ views on the social responsibility of scientists and engineers. 
The overall results revealed that pre-service teachers had a good awareness of ENVIR, CIVIC, COMMU, CONSEQ, 
COMGOOD, CONSEQ, and COMGOOD, while HUMAN, NEEDS, and POLICY areas were the lowest. Ko et al. (2022), 
who studied social responsibility with science and engineering students in South Korea, are very similar to the 
results in ENVIR, COMMU, CONSEQ, and COMGOOD. Our results are not comparable to those of Ko et al. (2022) 
regarding the HUMAN and CIVIC areas. In our study, in contrast to the results of Ko et al. (2022), the lowest aver- 
age values for the area HUMAN. 

The overall results demonstrate that the mean scores for interest in and consideration of human, environ- 
mental, and social impacts (ENVIR and CIVIC, COMMU, CONSEQ, and COMGOOD) tended to be higher than mean 
scores for areas directly related to social practice (NEEDS and POLICY). This result is very similar to the study of 
Ko et al. (2022). However, our study found the lowest mean scores for the HUMAN and higher mean scores for 
the CIVIC areas. Therefore, the results regarding HUMAN and CIVIC areas differ from Ko et al’s study (2022). In 
light of the results, the participants had the lowest mean scores regarding the areas that required social partici- 
pation and action in science and technology research. For example, the participants indicated the lowest mean 
scores in the policy area compared to the other areas. Such a finding appears in the study of Ko et al. (2002) for 
the same area. This result may be because Chinese pre-service teachers wanted to avoid being part of a process 
that requires engagement and participation in science and engineering research. The reason for these results 
may stem from cultural factors. Similar to our results, the other researchers (Canney & Bielefeldt, 2015; Ko et 
al., 2022) who studied social responsibility have reported similar results. For example, Canney and Bielefeldt’s 
(2015) results showed that engineering students agreed more strongly with the importance of basic engineer- 
ing skills and the ability of engineers to impact society positively. However, these same students agreed less 
regarding their personal or professional commitment to helping others. The agreement of participants in many 
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areas of social responsibility can be explained by the nature of the ethics of care theory first proposed by Nod- 
dings (1984) in the early 1980s. This theory accepts social responsibility as its second dimension (Sevenhuijsen, 
2003). According to this theory, social responsibility explains the willingness and capacity to take responsibility 
for doing something for a specific need. 

This study showed no significant differences regarding participants’ views of social responsibility across 
all subfactors (see Table 4). Part of this result is consistent with that of Ko et al. (2002), and the other part is not 
consistent with the study of Ko et al. (2022). They found significant differences in Human, Envir, and Conseq in 
favor of the female participants. Moreover, our results are not related to the studies suggesting the ethics of the 
theory with females (Bergmark & Alerby, 2006). For example, Bergmark & Alerby (2006) indicate that researchers 
have associated care ethics with females. 

The results regarding the grade levels of the participants showed no significant differences. This result is 
consistent with Ko et al. (2022), who found no significant differences among grades of science and engineering 
majors in all areas. This result is interesting because the participants who enrolled in the fourth grade of education 
faculty received some courses on the nature of science and science-technology-society. In these courses, they 
were taught about the impact of science and technology on society. However, the results showed no differences 
between grade levels. In light of these findings, taking some science and engineering education courses may 
produce a poor awareness of the social responsibility of scientists and engineers. 


Conclusions and Implications 


This study aimed to examine Chinese PSTs’ views on the social responsibility of scientists and engineers. 
Since more information is needed about how PSTs view the social impact of scientific and engineering research, 
what factors are influential in understanding social responsibility from their perspective, and who will teach 
future STEM majors in schools, there has been a research gap in the literature. Because of this reason, the present 
study was necessary. The results revealed that pre-service science teachers had high mean scores in five areas 
of social responsibility and a good awareness of the role of scientists and engineers in addressing problems 
and risks related to human, environmental, and societal impacts. However, the results also revealed lower mean 
scores in three areas requiring action or involvement. In addition, the results did not show statistical differences 
according to gender and grade level. Considering there has been no research on the views of PSTs on the social 
responsibility of scientists and engineers, the results obtained from this study present new insights into research- 
ers’ knowledge and add new findings to the literature. Thus, the present study contributes to the literature by 
adding new perspectives for researchers. In addition, the study has helped to fill a critical research gap in the 
literature. This study is an important starting point to emphasize the importance of socially responsible teachers 
in the science education literature. Future researchers should put more effort into understanding pre-service 
teachers and the underlying factors that shape their understanding of social responsibility. 


Recommendations 


The present study presents critical implications for pre-service science teacher education. First, this study 
presents important findings about the social responsibility of PSTs. The results have the potential to contribute to 
the literature. Future research should continue to consider social responsibility as part of the STEM education of 
prospective teachers. The results of this study allow researchers to draw some conclusions about teaching social 
responsibility in science and engineering research. In particular, a greater emphasis on science and technology 
research in a societal context could enable prospective science teachers to develop their understanding of the 
impact of science and technology on society and social problems. To enhance pre-service teachers’ understand- 
ing of the social responsibility of scientists and engineers, educators and scientists in teacher education should 
aim to provide prospective teachers with diverse experiences and learning opportunities about the societal 
impacts of science and technology. Teacher education that provides opportunities and courses that combine 
science, engineering, and social responsibility will result in socially responsible teachers who teach STEM in 
schools. The experiences and opportunities provided by educators and scholars to participate in a science and 
engineering research project or to observe the effects of science and technology developments will enhance 
pre-service teachers’ awareness and understanding of the impacts of science and technology on society and 
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social problems. To this end, pre-service teachers should be encouraged to participate in activities for everyone 
or students organized by professional associations of scientists and engineers. In addition, further research is 
needed to measure the impact of these types of activities on the social responsibility of prospective teachers. 

The findings suggest that prospective science teachers need to be encouraged to educate them about 
the positive impact of science and technology on society and social problems. In this way, science education 
can achieve its goal of having socially responsible teachers and STEM graduates. Educators need to teach social 
responsibility in their courses by emphasizing the impact of science and engineering on society, and their social 
impact should be emphasized in the education of science teachers. 

Second, because this research was conducted with a specific group of participants, future research 
must include diverse participants from different cultural and educational contexts. Given that the number of 
research studies on social responsibility and prospective science teachers is very limited, more research needs 
to be conducted with prospective science teachers in different countries and cultures. Third, the results showed 
that pre-service teachers are less fearful of participating in the research process in science and technology. 
Given the lack of educational interventions that emphasize the social responsibility of scientists and engineers 
in teacher education, tailored practical suggestions and educational programs are needed to be developed by 
educators and scholars. 


Limitations 


First, this research involved a limited number of PSTs. Second, a quantitative data collection method was 
used to answer the research question. Therefore, the results have some limitations related to the type of data 
collection instrument used. Although we used a newly developed and more comprehensive social responsibility 
instrument, additional data collection instruments may be considered by other researchers. The PSTs’ views of 
social responsibility may vary depending on participants’ backgrounds, cultural and educational contexts, and 
discipline. In light of these factors, our study may have some limitations. 
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Appendix 1. Comics produced by the study group 


DCB Nisa (continues) 


Fen édevim icin nesli tiikenen 
hayvanlardan birini arastirmam 
gerek, ama hangi 

hayvant ????? Buldum. 
Anadolu leopari olur. Zaman 
makinemle anadolu leoparinin 
yasadigi zamana gidebilirim. 


Merhaba. Sey 
ben buraya 
anadolu 
leoparlarini 
arastirmak icin 
geldim acba 
bana yardim 
eder misiniz ? 


J 


; Biraz 
0 o bir anadolu ilerlemeliyim. Elime 
leopari. Yasasin defter ve kalemimi de 
gelmisim. alayimki 6devimide 
yapabiliyim. 


Tabi Yaz boyu 200-250cm adirligi disilerde Yazdim. Cok 
ki olurum 35-40kg erkeklerde 55-70 kg’dir. tesekkirler. Artik 
benimle gel. Yaklasik mrti 20 senedir. Etcildir. eve dénsem iyi 

Hadi. Habitatlari yok olduklani icin olur, zaten hava da 
kéylere-kasabalara yonelmeye kararmaya basladi. 
basladilar. Bu da onlarin vurularak 
yada zehirlenerek dldiiriilmesine 
neden oldu .Bu nedenle nesilleri 
tiikenmeye basladi. 
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Tamam 
-Gériisiiriiz. Tamam 
Yine gel olur - Gelirim, 
mu? giile gille. 


Merhaba! 
Ben Nejdet. 5. 
sinif 
6Grencisiyim. 


ISSN 1648-3898 sprint 
ISSN 2538-7138 soniines 


Olamaz 
saat tam 
11.30.hemen 
yatmaliyim. 


Ben 
de gogu gocuk 
gibi atrstirmalik 
gidalar tuketmeyi 
seviyorum. 


Dondurma, 
cikolata, 
sekerleme ve sakiz 
sevdiklerim 
arasindadir. 


Bu 
atistirmaliklardan 
bir cogunu aslinda 
eriskinler de sikca 

tiketmekte. 


— 


Ben burada sakiz tiketimine 
6zel bir parantez acmak 
istiyorum. Ciinkii diger 
Uriinlerin aksine sakizin 
cevre acisindan seriiveni 
cignendikten sonra 
baslamaktadir. 


Alacagimiz 
cok basit 
dnlemlerle dogayla 
barisik olmak 
miimkin. 


Hayir! 

Tabi ki de 

ayakkabinin altina 
yapismasindan 


bahsetmeyecegim. 


Sakizin dogada 2-5 
yiida parcalandigint 
disiinirsek, sorun 
tahmin ettigimizden 
daha biiyuk. 


Asil problem, cignendikten 
sonra cevreye atilan 
sakizlarin kuslar tarafindan 
besin sanilarak tiketilmeye 
galisiimast sonucunda 
6liimlere yol acabilmesi. 
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Geyik kardes 5 insanlar 
keske bu kadar iyi kendince ¢ok iyi 
ve gizel bir yerde yastyorlar ama bizi 

yasasak. 


Buraya 
hep ¢6p atyorlar 
buna bir ¢ézim 
bulmaliyiz 


katiltyorum 

bunu insanlanin 
yardimiyla 
yapmaliyiz 


yardim 
etmezseniz 
gelecekte 
size patlar 


yadiminizi asla cok 


ederiz 


Evet 
oglum cok 
gizel. 


Oglum ben 
su ¢adin tophyayim 

« Bugiin son glinimiiz 
canim. 


Evet anne 
ya iste bu yizden 
cok Uzgiinum. Burasi 
cok giizeldi . 


Aa Anne 

bu hafta sonu 

kampa gidelim 
mi? 


acaba anneme 
sorsam yine kampa 
gider miyiz ? Hemen 
sorayim. 


tesekkiir 


Oof ya 
cok yoruldum bir 
an dnce eve gitmek 
istiyorum . 


haklisin 
kardesim 


Bilmiyorum 
glum 
yemekten sonra 
konusuruz . 


mi ? Yorulmus 


hadi kardesim 
baslayalimmm ikimizde ayrihp 
‘ozaman insanlardan yardim 
isteyelim 


Ah 
oglum geldin 


Evet 
anne ya sporda 
cok yoruldum biraz 
odamda 
dinlenicem. 


gibisin . 


Anne 
kamp isine Tamam 
oldu. oglum yemekten 
sonra konusuruz 
dedim ya . 


Imm 
acaba kamp. 
icin yanima neler 
alsam ? 


Yasasin O3glum 


kampa 


¢0k istiyorsan bu 
gidiyoruz . 


hafta sonu kampa 
gide biliriz 
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Anne Evet oglum 
ben burayt hic Bencede maalesef insanlar Anne 
begenmedim . oglum hemen atiklarini ormanlara ormanlanimiz 
Hemen eve eve gidelim . veya dogaya yok oluyor . 
gidelim . attiklart icin ve 
bilingsizce agaclari 
kestikleri icin gitgide 
doga yok oluyor . 


Evet sevgili dostlarim cizgi 
romanimi gérdilyseniz 
sizde dogamizi koruyun « 
Cinki bu béyle devam 
ederse dogamiz yok olacak 
. DOGANI KORU ... :) 


DCB Irem 


DCB Alper (continues) 


Neden Ormanlar Hig Ormanlanimizt Sehirin Ormanlaru yok 
buralarda hic yok oldugu i¢in yagmurda yok etmeseydik havasida cok edip yerine bina 
hayvan yok hayvanlanin yagmiyor ormanlar kirli neredeyse yapmasaydik cok 
barinacagi bir yer sayesinde nefes alinmiyor daha temiz bir 
kalmadi atmosferde havamiz olurdu. 
yagis ortami Ormanlar 
olusabilirdi diinyanin 
cigerleridir 
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Sagol 
Abdulrezakeim 
iyiyim bu havada 
yagmur yaamaz 
dedim ama... 


Herhaba 
Hocam.Nasilsiniz 


Iste 
bunlan yapar 
yapmayanlan uyanrsak 
hava kirliligini ortadan 
kalkar 


Simdi Hocam 

sira ses insanlar nasil bu 

kirliligine kadar gUriltuld bir 
ortamda 
yasiyor ? 


Bahse 
varalim ki 
Gogumuz okulu ve 
ders calismayi pek 
sevmiyoruz. 


CORRECTION ON USING COMICS FOR CLIMATE CHANGE IN SCIENCE EDUCATION: 


OGretmenim, 
size bir sorum var. 
Kiresel tsinma 
diinyamizi nasil 
etkiliyor? 


Géryor musun 
Abdulrezzakeigim 
bu havada ne 
olacagt hi¢ belli 
dea 

Aniden durdu 


Buzullar eriyor ve 
deniz seviyesi 


gittikce yilkseliyor . 


Buda kutup 
ayilarinin neslinin 
tukenmesine yol 
aciyor 


STUDENTS’ 


Peki 

kiresel jsinmayt 

dnlemek icin ne 
yapmamiz 
gerekiyor? 


Kendi arabamizdan 
¢ok toplu tasima 
kullanmah, agac¢ 
dikmeli ve giines 
enerjisi 
kullanmahyiz 


Evet hocam 
haklisiniz kiresel 
isinma yliziinden 
ne zaman 
yagmur yagacagi 
hig belli olmuyor 


EGer kiiresel 
1sinma bu 
sekilde devam 
ederse sonu 
tim caniilar 
icin cok Koti 
olacak 


Evet 
maalesef dogru 
séyliyorsunuz hocam 
acil 6nlem almalyiz 
ama nasil 


SOLUTIONS AND AESTHETIC SUBTLETIES 


(Ppp. 549-558) 


ilk Evet 
nce fabrika deodorant 
bacalarindan cikan araba egzozlan gibi 


gazlar filtrelenilmeli gazlari da filtre 
takilmali 


gelelim su 
kirliligine 


Cok haklisin bu 
bu yiizden kuslar 
seslerden dolayi 
geceyi sabah 
saniyor 
insanlarda isitme 
kaybina yol 
aciyor. 


Ogretmenim 
su kirlilige 
bakarmisiz cok 
pis! 


Zaten 
sesin fazia 
oldugu yerde 
kimse barinmaz 


Cok 
haklisin 
Abdulrezakcim.insan! 
ar bilingsizce su 
kaynaklarint 
tiketiyor... 


DCB Ada (continues) 
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Fakat 
okulun ve ders 
calismanin da 

faydalani vardir. 


Evet dGretmenim 
béyle giderse 2050 
yilinda diinyanin %40 
su sikintist cekecek 


lik nce gece ge¢ 
saatlerde yiksek 
seslerde sarkilar 
acilmamal yiksek 
sesli yerlerde 
caligan insanlar 
mutlake 
Snlemlerini almali 


Sanirim 
elektirikler 
kesildi. 


ilk dnce israftan 
kacinacaz 
dislerimizi 
fircalarken suyu 
agik birakmicaz 
kanalizasyonlani 
sulara 
akitmamacagiz 


feci bi durum bu 
durumdan kurtulmak 
igin ne yapmahyiz 


Eveeet 
Abdulrezzak 
c1g1m,bu 
anlathklarimiz 
nsaniara faydal 
olur. 


insallah hocam 
bizde bize 
birakilan doga 
mirasini,koruyup 
gelecek nesillere 
aktarabiliriz . 


Le 
AD 


Elektrigi 
bosa 
harcamamak cok 
Snemlidir. 


553 


504 


CORRECTION ON USING COMICS FOR CLIMATE CHANGE IN SCIENCE EDUCATION: 


STUDENTS’ SOLUTIONS AND AESTHETIC SUBTLETIES 
(PP. 549-558) 


Mesela kullanmadigimiz bir 
odanin si9ini kapatmak 
veya i2lemedigimiz bir 
televizyonu kapatmak icin 


simdi 
odadan 
‘cikacagim ve wsiGume 
da kapatacagim 


Journal of Baltic Science Education, Vol. 22, No. 3, 2023 


ISSN 1648-3898 | prrinty 
ISSN 2538-7138 /oniines 


Ellerimi 
yikadim fakat 
15191 
kapatmiyacagim. 
Gibi.... 


simdi 
de yanlls bir 
davranisa érnek 
verelim. 


DCB Yagmur 


Din gece 
yedigim meyve 
tabagint pek iyi 

yikamamim. Sebebi bu 

olabilir mi ? 


Eger 
atiklar meyvenin 
yetistigi toprakla 
birlestiyse, evet 

olabilir. 


Acaba 
neden 
olabilir ? 


Bana 
atiklann toprakla 
nasil birlesmis 
olabilecegini 
anlatabilir misin ? 


Sanayi atiklari 
toprage gémiiliince 
toprakla birlesir ve ya 
kirli su topraga 
dékillince toprakta 
yayllir ve TOPRAK 
KIRLILIGI ortaya 
gikar. 


Elbette 
anlatirim 
kizim ... 


O zaman 
ben meyveyi 
yikarken iserinde toprak 
kaldigt icin mi karnim 
agniyor ? 


Galiba evet 
Meliscim ama emin 
olmak icin doktora 
gériinmen gerek. 
Hadi gel hastaneye 
gidelim. 


Zehirlenmis 
olabilirsin. midende 
zararh bir atik 
gériiniyor, son 
zamanlarda yedigin 
bir sey sana 
dokunmus 

olabilir mi? 


Hepinize 
merhaba 
arkadaslar 


kiminle 
konusuyorsun 


Hosgeldiniz 
ben de sizi 
bekliyordum 


Bekledigini 
bilmeseydim 
kosmazdim 

zaten 


Evet, diin gece iyi 
yikanmamis bir 
meyve yedim 
ama galiba 
meyve kirli bir 
toprakta 
yetismisti. 


Bir 
daha yiyecegim 
seyi iyice 
yikamadan 
yemicegim. 


Su an ucuz atlattin 
sana bir hap vericegim 
yarina iyilesirsin ama 
bir dahakine dikkatli 
ol toprak kirliligi cok 
biiyik saglik 
sorunlarina yol 
acabilir. 


Bu 
sorumlulugu 
aldigina 
sevindim. 


Eyvah! 
Sakin bana 
Selen'in orada 
bekledigini 
sdyleme. 


parka gidiyoruz 
bugiin arastirma 
ginal 


Arkadaslar fen hocasinin 
verdigi ddev ile alakali olmak 
zere konumuz “Canhilarin 
yok olma sebepleri neler 
olabilir?”arastirmam izdan 
nce fikirleri alalim. 


Bence 
biyle bir 
sey olmaz 
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O zaman 
tazmanya 
kaplaninin nesli 
neden tikendi 
Zeynep 


Bunu 
toprak 
kirliligi 
olarak 
ifade 
edebiliriz 


Bir 
de bundan 
birden fazla 
oldugunu distin 
gordiin mii? 


olacak nefesi 
kesilir aaaa 


Azra sormadan ben cevabi 
veriyim hava kirliligi 
solunum yolu hastalikiarina 
neden olabilir 


STUDENTS’ 


Ohhaal 
burada 
amma 


var her 


Burast 


oldu y 


DCB Ela 


¢ok araba 


yer egzoz 


kalabalik 


dedistirelim 


Tamam 
da bunun canhilarin 
neslinin tikenme 
sebepleriyle ne ilgisi 
var 


hs 


cok 


er 


Bence 
diger 

parka 
gidelim 
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Séyle distin Azra diyelim 
ki biri sokaga sakiz atti ve 
bir kusta onu yemeye 
¢alisti nolur? 


Sanirim hava 
kirliligi bu da 
caniilarin 
neslinin 
tukenmesine 
neden 
olabilir 


Bunu da 
listeye 
ekleyelim 
o zaman 


ellerimizi 
yikayalim 


Abi 
su temiz 
akmiyor 


Evet cocuklar 
bir siredir 
yandaki fabrika 
céplerini suya 
dékiiyor o 
ylzden sular pis 


Ben 
bi abimi 
caginiyorum 


Pis 
su igersen ne 
olur Zeynep? 
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arkettiniz 
mi bir cok 
irlilik 
Insanlar 
tarafindan 
lusturuluyor 


O zaman 
kisaca su 
kirlligi var 


Evet bu ylizden 
insanlan 
bilinclendirmeliyiz 


Arkadaslar 
bugiin caniilarin 
yok olma 
sebeplerini 
inceledik eminim 
ki baska yok 
olma sebepleri 
de vardir 
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DCB Oguz 


Neden bu kadar ¢ok Bayle diistindugiind Mesela agaclar havayi 
Dogay: seviyorsun?Onlar yerine bilmiyordum.Doganin temizler, oksijen iretir, Vay 
ve agaclan binalar yapabiliriz, ciinki ve agaclarin bir siirii Olur erozyonu &nler, enerji canina! 
sok seviyorum. bir ise yaradiklarint faydalari var.istersen bahsedebilirsin. ve su tasarrufu yapar, Gercekten bir stir 
sanmiyorum. sana bunlarin bir Seni dinliyorum. sera etkisi ile miicadele faydalan varmis. 
kagindan eder, sebri ve sokaklari 
bahsedebilirim. serinletir, cocuklart 
ultraviyole isinlardan 
korur ve bunun gibi 
binlerce faydalani 
vardir, 


iste 
inis yaptik Evet 


Ve eger istersen - iste 
sana agaclanin Blinc ' . béyle \MAZ becierhn ; 
olmadigi bi es beer é ai TER tamam sakin of iyi misin? iyiyim ama bunu 
dinyann neye gbstereceksin crete all pUsUYORU S$ azkaldi simdi inis arene Beds yaptigint hala 
benzedigini peki? isinlanarak. yapacagiz. anlayamadim. 

gésterebilirim. y 


Ciinkii agaclar bizim iste simdi doganin ve agaclarin Rica 

oksijen kaynagimizdir ve ne kadar dnemli oldugunu ederim, artik Tabi 

agaclar olmadiginda anladim ve artik doga hakkinda dogay: daha cok ki, umarim benim 
oksijensiz kaliriz ve canh daha bilincliyim.Sana da beni bu sevdigini gibi diisiinen herkes 
yasami sona erer. konu hakkinda bilinclendirdigin dustinuyorum. bir giin bilinclenir. 
icin tesekkiir ederim. 


iste agaclanin 
olmadigt bir 


ri kadar kétii 
van vane géziikliyor ve neden 


4 burada hic bir canl 
kadar kétii varhk yok? 


Evet ne 


oldugunu 
gériyor musun? 


DCB Mete 


Halkim, ilkemizde her giin yizlerce 
Clamez, agac kesiliyor. Agaclar bizlerin ve diger Tam Aynen 

ook gee kaldent caniilarin yasamlarint stirdiirmeleri icin istedigimiz dyle orman 
Hemen baskana cok dnemlidir. Bu yiizden artik her gibi baskanim. /\ koruyucusu. 

haber vermetiyim. ormaniik alant dronlarla kontrol . 

edecedgiz. Bu sekilde agaclari kesenleri 

tespit edip cezasini verecegiz. Benimle 

bu ise var misiniz? 
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Appendix 2. Observation Form 


Observation Phase: 


Item 1: 
Getting support while using Pixton (resource 
books, teacher, friend, etc.) 


Item 2: 
Active participation in the learning process (asking 
questions, answering, commenting, etc.) 


Appendix 3. Document Review Form 


Student Digital Comics 


tem1: 
ntroduction knowledge 


tem 2: 
Creating a problem 


tem 3: 
Finding a solution 


Item 4: 
Artistic subtleties 


Appendix 4. Researcher Notes Form 


Student Digital Comics 


Overview 


Problem codes 


Solution codes 
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Appendix 5. Data Collection Process Photos 


This article is an erratum for “Alp, G., & Coskun Onan, B. (2023). Using comics for climate change in science education: Students’ 
solutions and aesthetic subtleties. Journal of Baltic Science Education, 22(2), 215-231. https://doi.org/10.33225/jbse/23.22.215". 
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